blob: 095dfc689fba5f4e7822f2aae73ae35556e52f46 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
|
# Pipeline generator
This tool is a pipeline generator that currently supports the following platforms: `Nifi`, `Telegraf`.
This was built as a part of my Bachelor's thesis.
## Usage
- `source venv/bin/activate`
- `pip install -r requirements.txt`
- Setup sutff in `config.py`, for more info look at [Configuration info](#Configuration-info)
- Then run `python main.py`
## Configuration info
Configuration variables are defined in `config.py`, where sample variables are also already defined.
WIP
### Variables
In interactive mode you can manually give the API url, credentials and data fields for the pipeline from API response
If you opt out the interactive mode you have to define needed variables in the config.py file before using the tool.
The following variables have to be defined if interactive mode is tured to `False`
| Variable Name | Description | Example Value |
|-------------------------------|-------------------------------------------|-----------------------------------------------------------------------------------------------|
| API_URL | API url that returens JSON data | `https://api.open-meteo.com/v1/forecast?latitude=58.38&longitude=26.72¤t_weather=true"` |
| API_FIELDS | Data fields for pipeline, with json paths | `{'temperature': '.current_weather.temperature', 'windspeed': '.current_weather.windspeed'}` |
| API_USERNAME | Api username, if required by API | `"Placeholder"` |
| API_PASSWORD | Api username, if required by API | `"Placeholder"` |
| PIPELINE_SCHEDULING_PERIOD | How often the pipeline should run | `"5 sec"` |
| PIPELINE_NAME | Name of the pipeline | `"test_pipeline.json"` |
#### Database
Database variales have to be filled out before using the tool as currently there is only influxDB support.
| Variable Name | Description | Example Value |
|---------------------|--------------------------------------|--------------------------------------------------|
| `DB_URL` | Url to the influxDB for the pipeline | `http://influxdb:8086/write?db=nifi_weatherData` |
| `DB_USER` | Database usernmae | `admin` |
| `DB_PASS` | Database passwod | `admin` |
#### Nifi
Here Nifi Host, Username and Password are needed for deploying the pipeline to the Nifi Host.
| Variable Name | Description | Example Value |
|---------------------|----------------------------------------------|----------------------------|
| `NIFI_HOST` | Base URL of the Apache NiFi instance | `https://127.0.0.1.nip.io` |
| `NIFI_USER` | Username for authenticating in Nifi | `nifi_username` |
| `NIFI_PASS` | Username for authenticating in Nifi | `nifi_password` |
| `NIFI_DEPLOY` | Automatically delpoy pipeline to Nifi | `True` |
| `INTERACTIVE_MODE` | If False you can parse through api call data | `True` |
## Current Repo structure
```
|-- common
| `-- core.py
|-- config.py
|-- main.py
|-- modules
| |-- nifi
| | |-- core.py
| | `-- templates
| | |-- ...
| `-- telegraf
| `-- core.py
|-- pipelines
| `-- <Generated pipelines>
|-- README.md
`-- requirements.txt
```
### Samples
Currently testing with 2 following apis (sample urls):
- https://api.open-meteo.com/v1/forecast?latitude=58.38&longitude=26.72¤t_weather=true
- https://delta.iot.cs.ut.ee/measurement/measurements?source=780&?dateFrom=2024-10-15T00:00:00Z&dateTo=2024-10-16T00:00:01Z&pageSize=200&type=KogEN
|
