How to setup Docker container with Spark NLP and PySpark

[maziyarpanahi]

This is a rough template I used to use to have Spark NLP, PySpark, Jupyter, and other ML/DL dependencies as a Docker image:

#Download base image ubuntu 18.04
FROM ubuntu:22.04

ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME /home/${NB_USER}

ENV PYSPARK_PYTHON=python3
ENV PYSPARK_DRIVER_PYTHON=python3

RUN apt-get update && apt-get install -y \
    haveged \
    tar \
    wget \
    bash \
    rsync \
    gcc \
    libfreetype6-dev \
    libhdf5-serial-dev \
    libpng-dev \
    libzmq3-dev \
    python3 \
    python3-dev \
    python3-pip \
    unzip \
    pkg-config \
    software-properties-common \
    graphviz

RUN adduser --disabled-password \
    --gecos "Default user" \
    --uid ${NB_UID} \
    ${NB_USER}

# Install OpenJDK-11
RUN apt-get update && \
    apt-get install -y openjdk-11-jdk && \
    apt-get install -y ant && \
    apt-get clean;

# Fix certificate issues
RUN apt-get update && \
    apt-get install ca-certificates-java && \
    apt-get clean && \
    update-ca-certificates -f;
# Setup JAVA_HOME -- useful for docker commandline
ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
RUN export JAVA_HOME

RUN echo "export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/" >> ~/.bashrc

RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN pip3 install --upgrade pip
RUN pip3 install jupyter notebook==6.* numpy pyspark==3.1.2 spark-nlp==3.4.3

# Make sure the contents of our repo are in ${HOME}
RUN mkdir -p /home/jovyan/tutorials
RUN mkdir -p /home/jovyan/jupyter

RUN jupyter notebook --generate-config
COPY jupyter_notebook_config.json /home/jovyan/.jupyter/jupyter_notebook_config.json
USER root
RUN chown -R ${NB_UID} ${HOME}
USER ${NB_USER}

WORKDIR ${HOME}

# Specify the default command to run
CMD ["jupyter", "notebook", "--ip", "0.0.0.0"]

and jupyter_notebook_config.json for the password:

{
  "NotebookApp": {
    "password": "sha1:65adaa6ffb9c:36df1c2086ef294276da703667d1b8ff38f92614"
  }
}

Hopefully, this can be a good start.

UPDATE: Ubuntu 20.04, PySpark 3.1.2, Spark NLP 3.4.3

[roger-yu-ds]

It doesn't work for me. Pip downloads different versions of the same package and finally

#13 44037.2     raise ResolutionTooDeep(max_rounds)
#13 44037.2 pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000
------
executor failed running [/bin/sh -c pip3 install --no-cache-dir notebook==5.* numpy pyspark==3.1.1 spark-nlp pandas scikit-spark scikit-learn scipy matplotlib pydot tensorflow==2.3.1 graphviz]: exit code: 2

[maziyarpanahi]

This is merely an example of what it takes to have pyspark/spark-nlp in Docker, I don't believe it is supposed to work by copy/paste. I will remove the extra python packages from the example to not install them if not needed and also set the version to spark-nlp==3.0.3.