Getting Started
Let's get rocking with Magniv 🚀
Using Magniv is the easiest way to build and deploy data applications, pipelines, or cron-like jobs. A Python decorator based orchestration library at the core, Magniv allows data scientists and developers to schedule arbitrary functions in one line. Monitoring, CI/CD, and compute management all come out of the box using the Magniv dashboard.
Installation
Magniv core can be found on pypi and installed with pip
pip install magniv
Once you have everything set up, take a look at the documentation for the magniv library and follow one of our tutorials.
Why Magniv?
Today’s data stack was built for yesterday’s software engineers. Data scientists deserve their own tools.
Magniv is for teams struggling to hire that unicorn data scientist with deep infrastructure experience or for teams scrambling to organize complex webs of cron tasks and Airflow DAGs. By offloading data infrastructure to Magniv, data teams can be built from only data scientists.
Data scientists need the ability to create, deploy, and maintain their data applications, without relying on support from software engineers. Responsibility handoffs waste time and inevitably priorities end up lost in the JIRA ether.
Use Cases
Magniv can be used for so many types of tasks, from traditional data science projects to one-off scheduled jobs. We love hearing about new ways people use Magniv so send us an email if you have a interesting use case!
We use Magniv internally for a lot as well. Once you have an easy orchestration platform, approximate scheduled solutions start to feel much better than month-long model tuning projects. Sometimes it's better to skip building a model to better detect fraud when you can just run an hourly Python job to close invalid accounts.
Here are some ideas that we have seen:
- Entity resolution in public datasets
- Regular Slack message alerts on user review sentiment
- Building a GitHub scraper (despite API rate limits)
- Pulling Twilio logs for daily user post-processing
- Fetching and analyzing a regularly updated public data source
- Detecting fraud or suspicious activity on a regular interval after-the-fact
- Lead scoring social network users and generating mutual friend lists
Core Features
- Open source core
- Github-integrated CI/CD
- Task failure email notifications
- Task run logging
- Container native
- Little scaffolding, one line decorator
- Abstract away complex infrastructure
- Job organization and management
- Easy productionization of jobs
- Visibility into deployed jobs
- Workspace collaboration
- Simple Artifact Store [coming soon]
- Automatic Data Catologue [coming soon]
Learn about Magniv
Please check out our tutorials and technical documentation using the sidebar.
Below are some recommended pages to check out:
- The Magniv
@taskdecorator - Creating your first workspace (Tutorial)
- Building a simple Slack Bot with Magniv (Tutorial)
- General FAQs
Running Magniv
Magniv can be used in our fully-managed environment or you can self-host the orchestration servers.
Self-hosted setups work well for teams that need tight control of infrastructure and security. Fully managed setups are perfect for teams that want to move quickly and not worry about how things work in the backend.
| Fully Managed | Self-Hosted | |
|---|---|---|
| Magniv Core | Yes | Yes |
| Export Tasks to Airflow | Yes | Yes |
| Magniv Dashboard | Yes | No |
| Github CI/CD Integration | Yes | No |
| Teams & Collaboration | Yes | No |
| Managed Infrastructure | Yes | No |
Fully Managed
To get started with the hosted version of Magniv create an account and connect your GitHub repo. Our Getting Started Tutorial details this process.
Self-Hosted
For self-hosted installations take a look at our documentation for exporting tasks to Airflow using Magniv Core.
Privacy
User authentication on the hosted Magniv dashboard is set up through your GitHub account. Your source code will remain completely private to you, any users you share it with, and internally for debugging purposes.
Magniv will never use your repositories for anything other than the workspaces you have created in our dashboard. Feel free to restrict the workspaces that Magniv can read using this link.
All workspace instances live in their own Magniv instance. Users are completely isolated from eachother and user workspaces live in different boxes. Individual tasks are also isolated from eachother in separate containers.