Installation

Post Installation

Instructions for LLM-based Mode (v3)

Instructions for Semantic Parsing Mode (v1 + v2)

1. NL4DV installs nltk by default but requires a few datasets/models/corpora to be separately installed. Please download the popular nltk artifacts using:
   
   python -m nltk.downloader popular

2. NL4DV requires a third-party Dependency Parser module to infer tasks. Download and install one of:

   1. Stanford CoreNLP (recommended):

      • Download the English model of Stanford CoreNLP version 3.9.2 and copy it to `examples/assets/jars/` or a known location.

      • Download the Stanford Parser version 3.9.2 and after unzipping the folder, copy the `stanford-parser.jar` file to `examples/assets/jars/` or a known location.

        Note: This requires JAVA installed and the JAVA_HOME / JAVAHOME environment variables to be set.

   2. Stanford CoreNLPServer :

      • Download the Stanford CoreNLPServer, unzip it in a known location, and cd into it.

      • Start the server using the below command. It will run on http://localhost:9000.

        java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -annotators "tokenize,ssplit,pos,lemma,parse,sentiment" -port 9000 -timeout 30000

        Note: This requires JAVA installed and the JAVA_HOME / JAVAHOME environment variables to be set.

   3. Spacy :

      • NL4DV installs Spacy by default but requires a model to be separately installed. Install the sample English model using:

        python -m spacy download en_core_web_sm

      • If you face the urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3 error while installing spacy models, as per this stackoverflow post, perform the following operations until there is a better fix:

        pip uninstall urllib3 && pip install 'urllib3<2.0'

Sample Code for LLM-based Mode

from nl4dv import NL4DV
#Your dataset must be hosted on Github for the LLM-based mode to function.
data_url="https://raw.githubusercontent.com/nl4dv/nl4dv/master/examples/assets/data/movies-w-year.csv" #paste your data URL

# Choose your processing mode LLM or parsing. Choose "gpt" for the LLM-based mode or "semantic-parsing" for the rules-based mode.
processing_mode="gpt"

#Enter your OpenAI key
gpt_api_key="[OpenAI KEY HERE]"

# Initialize an instance of NL4DV
nl4dv_instance = NL4DV(data_url=data_url, processing_mode=processing_mode, gpt_api_key=gpt_api_key)

# Define a query
query = "create a barchart showing average gross across genres"

# Execute the query
output = nl4dv_instance.analyze_query(query)

# Print the output
print(output)

Output

{
  "query": "create a barchart showing average gross across genres",
  "dataset": "https://raw.githubusercontent.com/nl4dv/nl4dv/master/examples/assets/data/cars-w-year.csv",
  "attributeMap": {"..."},
  "taskMap": {"..."},
  "visList": ["..."],
  "followUpQuery": false,
  "contextObj": null
}

"attributeMap" ▸

{
    "Worldwide Gross": {
        "name": "Worldwide Gross",
        "queryPhrase": ["gross"],
        "inferenceType": "explicit",
        "isAmbiguous": false,
        "ambiguity": []
    },
    "Genre": {
        "name": "Genre",
        "queryPhrase": ["genres"],
        "inferenceType": "explicit",
        "isAmbiguous": false,
        "ambiguity": []
    }
  }
}

"taskMap" ▸

{
    "derived_value": [
        {
            "task": "derived_value",
            "queryPhrase": "average",
            "operator": "AVG",
            "values": [],
            "attributes": [
                "Worldwide Gross"
            ],
            "inferenceType": "explicit"
        }
    ]
}

"visList"▸

[
    {
        "attributes": [
            "Worldwide Gross",
            "Genre"
        ],
        "queryPhrase": "barchart",
        "visType": "barchart",
        "tasks": [
            "derived_value"
        ],
        "inferenceType": "explicit",
        "vlSpec": {
            "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
            "mark": {
                "type": "bar",
                "tooltip": true
            },
            "encoding": {
                "y": {
                    "field": "Worldwide Gross",
                    "type": "quantitative",
                    "aggregate": "mean",
                    "axis": {
                        "format": "s"
                    }
                },
                "x": {
                    "field": "Genre",
                    "type": "nominal",
                    "aggregate": null
                }
            },
            "transform": [],
            "data": {
                "url": "https://raw.githubusercontent.com/nl4dv/nl4dv/master/examples/assets/data/cars-w-year.csv",
                "format": {
                    "type": "csv"
                }
            }
        }
    }
]

Visualization (when the Vega-Lite spec is rendered) ▸

Sample Code for Semantic Parsing Mode (Single Turn Conversational Interaction)

from nl4dv import NL4DV
import os

# Initialize an instance of NL4DV
# ToDo: verify the path to the source data file. modify accordingly.
nl4dv_instance = NL4DV(data_url = os.path.join(".", "examples", "assets", "data", "movies-w-year.csv"))

# using Stanford Core NLP
# ToDo: verify the paths to the jars. modify accordingly.
dependency_parser_config = {"name": "corenlp", "model": os.path.join(".", "examples","assets","jars","stanford-english-corenlp-2018-10-05-models.jar"),"parser": os.path.join(".", "examples","assets","jars","stanford-parser.jar")}

# using Stanford CoreNLPServer
# ToDo: verify the URL to the CoreNLPServer. modify accordingly.
# dependency_parser_config = {"name": "corenlp-server", "url": "http://localhost:9000"}

# using Spacy
# ToDo: ensure that the below spacy model is installed. if using another model, modify accordingly.
# dependency_parser_config = {"name": "spacy", "model": "en_core_web_sm", "parser": None}

# Set the Dependency Parser
nl4dv_instance.set_dependency_parser(config=dependency_parser_config)

# Define a query
query = "create a barchart showing average gross across genres"

# Execute the query
output = nl4dv_instance.analyze_query(query)

Sample Code (Multi Turn Conversational Interaction)

Use the below Sample Code to get started with Conversational Interaction. A more detailed sample file can be found here.

Applications

Follow these steps to run the example applications:

• Download or Clone the repository using
  
  git clone https://github.com/nl4dv/nl4dv.git

• cd into the examples directory and create a new virtual environment.

  virtualenv --python=python3 venv

• Activate it using:

  source venv/bin/activate (MacOSX/ Linux)

  venv\Scripts\activate.bat (Windows)

• Install dependencies.

  python -m pip install -r requirements.txt

• Manually install nl4dv in this virtual environment using one of the above instructions.

• Run python app.py.

• Open your favorite browser and go to http://localhost:7001. You should see something like:

For the Jupyter Notebook application,

• cd into the examples directory.

• Install and enable the Vega extension in the notebook using

  • jupyter nbextension install --sys-prefix --py vega

  • jupyter nbextension enable vega --py --sys-prefix

• Launch the notebook using jupyter notebook.

  Make sure your Jupyter notebook uses an (virtual) environment that has NL4DV installed. Go to examples/applications/notebook and launch Single-Turn-Conversational-Interaction.ipynb to run the demo that showcases NL4DV's single-turn (standalone) conversational capabilities or Multi-Turn-Conversational-Interaction.ipynb for viewing NL4DV's follow-up capabilities

NL4DV exposes a simple, intuitive API for developers to consume. The below methods can be called on the nl4dv_instance object that is created after initializing NL4DV like nl4dv_instance = NL4DV().

{
"dialog_id": "0", "query_id": "0",
"attribute": {"medals": "Gold Medals"},
"value": {"hockey": "Ice Hockey", "skating": "Speed Skating"}
}
                                                

{"MPG": ["miles per gallon"],
"Horsepower": ["hp"]}

{"synonymity": 95,
"string_similarity": 85}

{"put": [("addition", "add")],
"add": [("addition", "add")]}

"also": [("also", "add")],
"as well": [("aswell", "add")]}

{'attribute': {
    'attribute_exact_match': 1,
    'attribute_similarity_match': 0.9,
    'attribute_alias_exact_match': 0.8,
    'attribute_alias_similarity_match': 0.75,
    'attribute_synonym_match': 0.5,
    'attribute_domain_value_match': 0.5,
},
'task': {
    'explicit': 1,
    'implicit': 0.5,
},
'vis': {
    'explicit': 1
}}

{"Year": "T", # Temporal
"Rank": "O", # Ordinal
"Salary": "Q", # Quantitative
"Gender": "N" # Nominal}

{"name": "corenlp-server",
"url": "http://192.168.99.102:9000"}

NL4DV can be installed as a Python package and imported in your own awesome applications!

1. NL4DV is written in Python 3. Please ensure you have a Python 3 environment already installed.

2. Clone this repository (master branch) and enter (`cd`) into it.

3. Create a new virtual environment.

   virtualenv --python=python3 venv

4. Activate it using:

   source venv/bin/activate (MacOSX/ Linux)

   venv\Scripts\activate.bat (Windows)

5. Install dependencies.

   python -m pip install -r requirements.txt

6. make your changes>

7. Bump up the version in setup.py and create a Python distributable.

   python setup.py sdist

8. This will create a new file inside **nl4dv-*.*.*.tar.gz** inside the dist directory.

9. Install the above file in your Python environment using:

   python -m pip install <PATH-TO-nl4dv-*.*.*.tar.gz>

10. Verify by opening your Python console and importing it:

    $python
>>> from nl4dv import NL4DV

11. Enjoy, NL4DV is now available for use as a Python package!

NL4DV was created by Arpit Narechania, Arjun Srinivasan, Rishab Mitra, Alex Endert, John Stasko of the Georgia Tech Visualization Lab, along with Subham Sah and Wenwen Dou of the UNC Charlotte Visualization Center.

We thank the members of the Georgia Tech Visualization Lab for their support and constructive feedback.

2024 IEEE VIS NLVIZ Workshop Track

@misc{sah2024generatinganalyticspecificationsdata,
    title={Generating Analytic Specifications for Data Visualization from Natural Language Queries using Large Language Models},
    author={{Sah}, Subham and {Mitra}, Rishab and {Narechania}, Arpit and {Endert}, Alex and {Stasko}, John and {Dou}, Wenwen},
    year={2024},
    eprint={2408.13391},
    archivePrefix={arXiv},
    primaryClass={cs.HC},
    url={https://arxiv.org/abs/2408.13391},
    howpublished = {Presented at NLVIZ Workshop, IEEE VIS 2024},
    }

2022 IEEE VIS Conference Short Paper Track

@inproceedings{mitra2022conversationalinteraction,
  title = {Facilitating Conversational Interaction in Natural Language Interfaces for Visualization},
  author = {{Mitra}, Rishab and {Narechania}, Arpit and {Endert}, Alex and {Stasko}, John},
  booktitle={2022 IEEE Visualization Conference (VIS)},
  url = {https://doi.org/10.48550/arXiv.2207.00189},
  doi = {10.48550/arXiv.2207.00189},
  year = {2022},
  publisher = {IEEE}
}

2021 IEEE TVCG Journal Full Paper (Proceedings of the 2020 IEEE VIS Conference)

@article{narechania2021nl4dv,
title = {{NL4DV}: A {Toolkit} for Generating {Analytic Specifications} for {Data Visualization} from {Natural Language} Queries},
shorttitle = {{NL4DV}},
author = {{Narechania}, Arpit and {Srinivasan}, Arjun and {Stasko}, John},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
doi = {10.1109/TVCG.2020.3030378},
year = {2021},
publisher = {IEEE}
}

If you have any questions, feel free to open a GitHub issue or contact Arpit Narechania.

The software is available under the MIT License.