Diskover Data Management Platform
Core Features
Take Control of ALL Your Data with Diskover
Diskover’s Core Features
The Diskover Data Management Platform is like a Swiss army knife offering various integrated tools, as well as smart plugins. At the heart of it all, Diskover “connects” all your data storages and file systems giving you global visibility via your favorite web-browser.
Industry Specific Editions
While this page summarizes Diskover’s core features, additional tools are available with these industry specific editions:
➡ Media & Entertainment Edition
➡ Life Science Edition
Your first step towards taking control of ALL your data.
Get hands-on experience with the Diskover platform with minimal to no deployment effort by leveraging the Diskover Community Edition within the AWS Marketplace. The Diskover Community Edition is free to use for an unlimited time, however applicable AWS EC2 instance resource charges apply.
Integrated Core Features Overview
Integrated SEARCHES
- Global view via any web browser > a single query will search all your data repositories connected to Diskover.
- Built-in search tools can be combined with manual queries for granular searches.
- Extra search options via business context metadata.
- Export and share your results in one click.
Integrated ANALYTICS
- Multiple standard and customizable reports for informed data management decision-making.
- Customizable storage cost reporting.
- Heatmap report compares data from two different points in time, monitors growth, shrinkage, and smooth transitions/backups.
Integrated ACTIONS
- Multiple integrated core and industry specific plugins.
- File action for one-click access to third-party platform, live view of directory, data movement, etc.
- Customers can deploy their own plugins for custom in-house workflows.
Integrated WORKFLOWS
- Automated data curation based on rules and rich metadata.
- Multiple plugins allowing for data curation, movement, integrity, actions, etc.
- Automated and manual tagging to facilitate data curation.
Flexible, Fast, SCALABLE
- Can index massive amounts of data at blazing speed, no matter where the data is located.
- Non-proprietary therefore ensures the safety of your source digital assets.
- Extra metadata harvesting adds business context to all aspects of data management.
Sustainable Solution BENEFITS
- Sustainable data management solution allowing you to reuse your data storage space and curate your data.
- Empowers all stakeholders, increases productivity, and reduces human errors.
- Reduces data-related operating costs.
Platform OVERVIEW
Integrated SEARCHES
Overview
- GLOBAL SEARCH: Diskover searches ALL your volumes/file systems and their directories at once, although you can limit your searches to a single path if needed.
- LIVE VIEW: The live view feature allows users to access a read-only version of the live directories. Users can drill down and/or search freshly onboarded data pre-indexing, as well as proactively copy paths.
- BUILT-IN SEARCH TOOLS: Diskover offers built-in tools like filters and quick search, which can be combined with granular manual queries for maximum efficiency.
- EXPORT/COPY: Users can export full sets of metadata in one click (json, csv) as well as copy/paste paths.
- SHARE: With a single click, users can share their search results with co-workers by sending a simple url link.
- SAVE QUERIES: Users can save personal queries so they can be re-launched at any time.
- GRANULAR SEARCHES: Search rules and syntax are based on Elasticsearch algorithms, which are extremely powerful and reliable.
Global View of ALL Your Data
Built-In Search Tools
Rich Metadata
Integrated ANALYTICS
Reports
Overview
Reports were designed to easily help you find your top unknowns. Diskover uses your rich metadata harvested during indexing to help you understand, analyze, automate, and monitor your data, therefore, giving you complete control to sustainably manage your digital assets.
This new analytical tool is customizable with what matters to your organization, from simple file types analysis to queries using business context. Filters can also be applied by any users for further granular investigation. This report is accessible to everyone in your organization. It is also global but can be limited to a specific path if desired.
Example Using Xytech Order Status
After creating the desired queries, select 1. the data you want to analyze, and 2. the number of top results you want in your comprehensive overview. In the example below, we are using Xytech order status metadata*. Note that any links under “by number” or “by size” can be opened to list the results by line items.
Still using the example below and following the organization’s guidelines, a decision could easily be made to free a significant amount of storage space by moving the “invoiced” data to a cost-effective asset preservation storage platform or simply deleting it after delivery to the client. Please note that this process can also be automated using Diskover’s integrated workflows.
Please note that the Reports analytic tool is included with Diskover Professional, Enterprise, Media Edition, and Life Science Edition.
Quick Reports by Volume
Executive Dashboard
File Tree Report
Treemap Report
Storage Cost Analysis
Overview | No Bytes Left on the Table
Diskover Data is leading the data management industry with its cost analysis tools and their degree of customization, therefore maximizing the ROI of your valuable assets.
Diskover offers tools to assist you determining storage cost for your clients’ data/projects, as well as detailed reporting to substantiate your invoicing.
When calculating the cost per gigabyte, companies should also consider compounding additional factors like electricity, storage providers’ cost, square footage of building space, support contracts, system administrator’s salary, Diskover’s annual subscription cost, etc.
Storage Cost Configuration Overview
Storage Cost Analysis Report
Space and Cost Summary per User/Group
Heatmap Report
Overview | Indices Comparison
Diskover Data is leading the industry by offering indices comparison from two different points in time, empowering you to easily track changes in your data. The heatmap report allows you to analyze and monitor data growth (red) or data shrinkage (green). The absence of colors is also a powerful indicator and desired in the case of smooth data migration or backup.
Heatpmap Report – Data Growth
Heatpmap Report – Data Shrinkage
Selection of Indices for Comparison
Indices Comparison from the Search Page
Power in History
“Having Diskover run and scan your disk is great and all, but if you wanted to merely see the size of folders you could use any disk analyzer software. Where it really starts to shine is the ability to compare disk scans against each other. Once you have two Diskover runs in Elasticsearch, Diskover will allow you to select two indices to compare. By default, each scan is timestamped to easily keep track of the history.
When comparing these, Diskover will now present you with additional features, such as heatmaps, to help you find which directories have the most fluctuation in size, either growing or shrinking.”
– linuxserver.io blog 06/28/2019
Smart Searches
Overview | Fully Customizable Queries
The Smart Searches report gives you the flexibility to build custom queries for easily repeatable reporting, for example, by project, customer, aging, size, owner, etc., as well as a mix of criteria for precise information finding and informed decision making.
This report is accessible to everyone in your organization and is global but can be limited to a specific path if desired.
Smart Searches Report
Integrated WORKFLOWS
Automated Scheduled Tasks
Overview
Diskover’s ultimate goal is global data curation through the various tools integrated in its data management platform. Diskover allows for processes to happen without manual intervention, therefore increasing productivity, as well as reducing human prone errors.
Diskover offers tools to sustainably solve the current data growth explosion, avoiding organizations to just buy more and more storage space, instead of organizing/cleaning their data through workflows.
Although all projects are unique, the production process remains the same. Via configurable plugins and automated scheduled tasks, Diskover makes data move through a virtual conveyor belt, allowing you to keep your data organized and reuse your existing storage space.
Due to its open-source infrastructure, Diskover allows for limitless integrations in order to automate workflows.
Far Beyond Data “Cleaning”
Diskover offers a full toolset of configurable scheduled tasks allowing for safe and controlled data curation, based on your organization’s rules, therefore allowing for storage space reutilization and assets preservation.
Increased Productivity Through Automation
Dataflow Example
Tags | Manual and Automated
Overview
Tags are crucial for successful data management and efficient workflows. In addition to manual tagging, Diskover offers plugins allowing scheduled automated tagging which are configurable based on your organization’s rules, as well as a tag copier plugin so tags are copied from one index to the next.
Tags Report
Automated Tags
Auto tags through scheduled tasks can be configured for automated data curation based on aging, project status, service contract agreement, etc.
Manual Tags
Diskover allows for manual tagging, alerting for actions to be taken, project workflow stages, as well as ease of searchability.
Integrated ACTIONS
Plugins Overview
Overview
Due to its open-source infrastructure, Diskover allows for limitless extensibility. Our solution offers various integrated configurable plugins with many more in development. This section offers an exclusive list of plugins, please click here to visit our SOLUTIONS page for the list of all major features.
Functionality Plugins
These plugins cover a wide range of functionalities like data management, curation via workflows, data movement, data integrity, etc.
File Action Plugins
The File Action plugins can be launched in one click from the user interface. They offer diverse functionalities like the live view of pre-indexed data, seamless access to third-party platforms, and data movement just to name a few.
Customer Deployed Plugins
Diskover allows for users to easily develop their own plugins to automate their custom in-house workflows. These plugins can be launched in one click using Diskover’s File Action feature.
Core Plugins
- Listed alphabetically
- CE = Community Edition, ESS = Essential, PRO = Professional, ENT = Enterprise, ME= AJA Diskover Media Edition, LSE = Life Science Edition
| PLUGIN | SHORT DESCRIPTION | CE | ESS | PRO | ENT | ME | LSE |
|---|---|---|---|---|---|---|---|
| Autoclean | Functionality | Designed to move, copy, delete, rename, or run custom commands on files and/or directories based on a set of highly configurable criteria. Any Elasticsearch query (tags, age, size, path, filename, etc.) can be used for the criteria providing very granular actions. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Auto Tag | Functionality | Designed to auto tag an existing completed index. Auto-tagging can also be done during crawl time by adding tag rules in the config file. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Checksums | Functionality | Adds checksums (xxhash, md5, sha1, sha256) to files in Elasticsearch indices. | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Checksums Hash Diff | File Action | Designed for precise data movement monitoring, it checksums xxhash, md5, sha1, and sha256 hash values between the original file and the resulting file once it reaches its transfer destination, alerting on areas where checksums don’t match. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Checksums S3 | Functionality | Adds checksums (md5, sha1) using AWS Lambda/Fixity to files in Elasticsearch indices built using S3 alt scanner. | ✔ | ✔ | ✔ | ✔ | ||
| Duplicates (Dupes) Finder | Functionality | Checks for duplicate files across a single or all indices using xxhash, md5, sha1, and sha256 checksums. Learn more. | ✔ | ✔ | ✔ | ✔ | ✔ | |
| Elasticsearch Field Copier | Functionality | Migrates Elasticsearch field data from one index to another. | ✔ | ✔ | ✔ | ✔ | ||
| Elasticsearch Query Report | Functionality | Sends email reports based on Elasticsearch search queries. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Illegal File Name | Functionality | Analyzes the index of all directories and file names for illegal characters, and long filenames or file paths to proactively find potential files with names that can break applications. Offending filenames are tagged with the corresponding non-conformance and the list of illegal filenames can then be sent via email reports. The plug-in can be configured to remediate these issues with automatic renaming or character replacement. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Index Differential | Functionality | Designed to provide a list of file differences between two indices (or points in time). The differential list can be used to feed synchronization tools or identify deltas where two repositories should be identical. Outputs a CSV file containing the differences between the two indices. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Live View | File Action | Allows read-only access to the live directories to search or drill down freshly onboarded data pre-indexing, as well as proactively copy paths. Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Qumulo Data Mover | File Action | Allows authorized users to move data from Qumulo to Qumulo, Qumulo to AWS, and AWS to Qumulo. Watch demo. | ✔ | ✔ | ✔ | ✔ | ||
| Tag Copier | Functionality | Designed to migrate tags from one index to the next. Generally, these tags are applied post index through manual tag application or plugin tag application (harvest, duplicate hashes, etc.). Learn more. | ✔ | ✔ | ✔ | ✔ | ||
| Unix Permissions | Functionality | Adds the Unix permissions of each file and directory to the Diskover index at time of indexing. Two tags are added, unixperms-plugin and ugo+rwx, if a file or directory is found with fully open permissions (777 or 666). Learn more. | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
| Windows Attributes | Functionality | Adds Windows file owner, primary group, and DACL info to files and directories in Elasticsearch indices. | ✔ | ✔ | ✔ | ✔ | ||
| Windows Owner | Functionality | Adds the Windows file owner and primary group of each file and directory to the Diskover index at time of indexing. It replaces all docs showing username 0 with the Windows file/directory owner name. Learn more. | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
New Core Plugins | Qumulo Data Mover
Diskover Data is thrilled to announce the Qumulo Data Mover plugins, allowing authorized users to manually trigger data movement from Qumulo to Qumulo OR Qumulo to/from AWS.
The process is as simple as finding the data you want to move and then launching File Action in one click to start the data movement process.
Both Qumulo and Diskover offer multiple integrated tools to monitor your digital assets migration and behavior. For extremely precise data integrity, the Diskover Hash Differential Plugin is designed to checksum hash values between the original file and the resulting file once it reaches its transfer destination, from Qumulo on-prem to AWS for example, catching and alerting on any possible file corruption in the process.
The first 2.5 minutes of this video summarize the data movement process and the monitoring tools. The rest of the video offers an overview of each platform and the supercharged infrastructure resulting from combining both Diskover and Qumulo.
Exclusive Plugins for Media Edition
Listed alphabetically. Please visit the AJA Diskover Media Edition page for all details.
| PLUGIN | SHORT DESCRIPTION | Learn More | Watch Demo |
|---|---|---|---|
| CineViewer Player | File Action | Seamless access to third-party platform allowing users to launch the CineViewer Player to view/validate media files, gives visibility to end users without access to the source asset. | Read | Watch |
| File Sequence | File Action | Designed to list any file sequences in a directory or from a single file in a sequence. | Read | |
| IMF Package Validator | File Action | IMF packages can be scanned and validated before delivery, from any location, regardless of the data location of the IMF package. | Read | Watch |
| Media Info Harvest | Functionality | Adds business context and searchability via additional media file attributes (resolution, codec, etc.). The enriched metadata is key for granular analysis, workflow automation, and data curation. | Read | |
| Telestream GLIM | File Action | Seamless access to third-party platform allowing users to launch GLIM to view/validate media files, gives visibility to end users without access to the source asset. | Read | Watch |
| Telestream Vantage | File Action | Seamless access to third-party platform allowing users to submit files to Vantage for transcoding directly from the Diskover user interface. | Read | Watch |
| Xytech Asset Creation | Functionality | Adds business context and searchability following assets rehydration via asset ID attributes. The enriched metadata is key for granular analysis, workflow automation, and data curation. | Read | Watch |
| Xytech Order Status | Functionality | Adds business context and searchability using order status enriched attributes (order phase, invoice date, etc.). The enriched metadata is key for granular analysis, workflow automation, and data curation. | Read | Watch |
⭐️ Customer Testimony – Vantage Workflow Integration at Visual Data Media Services
“Diskover has the ability to work with other applications if they have an SDK available. In the case of Telestream’s Vantage transcoding workflow platform, we have found that a combination of Diskover search tools and Vantage makes a really good team.
To eliminate the need to give users access to production content, and the ability to search through thousands of files, the Diskover team built a Vantage submit tool that directly allows to search a file, select that specific file or multiple files, and submit them to the Vantage workflow of your choice.
It’s extremely versatile and communicates very quickly. This procedure is only in its first phase so expanding it to allow workflow status submission, and more advanced workflows like audio overlay and subtitle overlays is on the road map.”
Randall Derchan
IT Manager, Visual Data Media Services
Once “submit to Vantage” is selected from File Action, the Vantage software opens, ready for a workflow to be submitted.
Exclusive Plugins for Life Science Edition
Listed alphabetically. Please visit the Life Science Edition page for more details.
| PLUGIN | SHORT DESCRIPTION | Learn More | Watch Demo |
|---|---|---|---|
| BAM Plugin | Functionality | Allows for the curation of genome sequence file transformation. | Read | Watch |
| Grant Plugin | File Action | Assist research institutes in managing their grants/members/storage costs internally, and fulfills the requirements for the new NIH DMS Policy. | Read |
Flexible, Fast, Scalable
Unique Indexing Architecture
If you can connect it, Diskover can index it.
- Diskover continuously scans all disconnected data repositories, in parallel not serially, and creates new indices so you can search, analyze, compare indices from different points in time, and manage your data in one global view via any web browser.
- Diskover offers alternate scanners, ex: S3 buckets, Dropbox, Microsoft Azure Blobs.
- Users can develop their own alternate scanners.
- Users can index offline media devices.
Offline Media Indexer
Access to the Live Directory
Via Diskover’s File Action features, users can securely access the live directory, proactively search and copy paths.
Maximum Efficiency Via Optimized Cached Scanning
Diskover customers using optimized cached scanning are reporting that, on average, scans take between 50% to 75% less time to build subsequent indexes. Note that results may vary depending on your infrastructure and storage type. This feature can be found in the Diskover annual subscription editions and is not included in the Community Edition.
Speed and Scalability
Overview
- Back ended by Elasticsearch, Diskover offers exceptional speed, reliability, flexibility, scalability, and handles massive amounts of data. Elasticsearch is used and trusted by giants like eBay, Airbnb, Uber, Shopify, Adobe, etc.
- Diskover efficiently and quickly searches distributed data scattered in different file systems, on-prem storage, cloud providers, offline devices, etc. bringing you all the answers in one global view.
- Well-suited data management solution for long-term asset preservation.
- Diskover is easily accessible via your favorite web browser.
- Supports all known file types.
🚫 Not a Storage System = Non-Proprietary
Diskover is non-proprietary. We do not store your files, we simply index their metadata. Therefore, users have access to read-only index of files and not the files themselves. Hence, Diskover gives your valuable digital assets the protection they need.
The Unlimited Potential of Open-Source
Diskover uses Elasticsearch, an open-source highly scalable, mature and wide spread backend search engine, allowing for massive amounts of data to be searched instantaneously. Elasticsearch also has a crucial role in indexing a 100% open-source platform allowing for unlimited expansion and plugins integration.
Diskover embraced the logical approach of itself being open-source, as well as integrating other open-source platforms and plugins that are powerful and highly integrative. Open-source software is developed around transparency and trust. They help organizations become more agile and collaborative in their innovations.
Become a Stargazer on GitHub
Diskover had its debut and grew tremendously on GitHub. The GitHub platform and its community will always remain very dear to us.
Join the thousand of Stargazers on GitHub following and contributing to Diskover.
➡ Download the free Diskover Community Edition
Sustainable Solution BENEFITS
Reduces Data-Related Operating Costs
The Cost of Unmanaged Data
- Are you continually buying additional storage instead of managing and reusing your existing space?
- How many daily man-hours are spent collectively on searching for data alone?
- Can you globally or granularly analyze all your data in order to take informed data-related decisions?
- Do you know where all your sensitive data is located therefore not leaving you exposed to data breaches?
- Are you unable to scale your business because of untenable data growth?
Data Management Has Become Inescapable
- Diskover is a cost-efficient solution proven to help significantly slash data-related expenditures. Take the first step by indexing your massive amounts of data at blazing speed.
- Diskover’s multiple tools and features assist all stakeholders in making the right data-related decisions about time, resources, invoicing, and investments.
- Diskover allows for global data management, therefore reusing storage space instead of purchasing additional expensive space.
Increases Productivity via Stakeholders Empowerment
- Through its ease of use and multiple features, Diskover improves collective efficiency.
- Via its global index, Diskover enables everyone to accurately answer questions with the simplicity of search and various analytics.
- The clever views, file search tools, repeatable reports, and other smart features assist all lines of business users for in-depth analysis of data, informed decision-making, as well as rigorous data management.
- In addition to cutting several man-hours, our scheduled automated tasks reduce manual interaction/human-prone errors and allow for safe and methodical data curation.
- Diskover offers multiple online resources and support to ensure the success of all its users.
