Visualizing Modulair-PM Data

silasblack · August 1, 2023, 7:46pm

Hi!

I was curious if there is a way through python to create this type of plot using data in .csv format pulled from a MODULAIR-PM consoles that was sampling off of Scripps Pier in San Diego, CA this month.

I attempted to follow the py-smps documentation (py-smps/examples/Visualize SMPS Data using py-smsps.ipynb at master · quant-aq/py-smps · GitHub), but reached errors when trying to install the py-smps library.

I’ve attached data in .csv format that I’m trying to plot in “histogram” format (not sure if this classifies as a histogram);

I want to use the columns ‘bin0’ through ‘bin23’ of data and plot essentially particle diameter vs concentrations for 24hr, 12hr and 6hr periods of time. I want to do it a similar if not the same way as this figure does (although I understand if there are limitations) and also be able to overlay 4 different graphs on top of each other (if possible).

Another representation I’m interested in making is this heatmap plot:

This would be a better way (in my opinion) to look at the overall data, date = x, diameter = y, normalized conc = z

I’ve been trying to work with matplotlib, pandas and numpy but I have little to zero coding experience and this is the best I could come up with:

Any help with this through python or otherwise would be greatly appreciated!

Thanks,

Silas

p.s. if the images I intended to include fail to display here, the vital ones are available at the py-smps github link pasted earlier

dmcc · August 1, 2023, 9:43pm

Thanks for the question, Silas! If I’m understanding correctly, py-smps can do the types of plots you’re hoping to make. Can you paste in the commands you ran/errors you were getting when trying to install py-smps? (and maybe some information about your OS?)

Thanks!
David

silasblack · August 3, 2023, 10:47am

Hi David,

I’m working from an HP Pavilion with Windows 11.

I’ve only been using the github documentation linked on the QuantAQ website and in my previous post; the example graphs I referenced above were taken directly from there.

Here is one example of an error message I get when attempting to ‘import smps’ using Jupyter notebook:

StopIteration Traceback (most recent call last)
File ~\anaconda3\Lib\importlib\metadata_init_.py:563, in Distribution.from_name(cls, name)
562 try:
→ 563 return next(cls.discover(name=name))
564 except StopIteration:

StopIteration:

During handling of the above exception, another exception occurred:

PackageNotFoundError Traceback (most recent call last)
Cell In[3], line 1
----> 1 import smps

File ~\anaconda3\Lib\site-packages\smps_init_.py:8
5 from .models import *
6 from .rcmod import *
----> 8 version = version(“py-smps”)

File ~\anaconda3\Lib\importlib\metadata_init_.py:1008, in version(distribution_name)
1001 def version(distribution_name):
1002 “”“Get the version string for the named package.
1003
1004 :param distribution_name: The name of the distribution package to query.
1005 :return: The version string for the package as defined in the package’s
1006 “Version” metadata key.
1007 “””
→ 1008 return distribution(distribution_name).version

File ~\anaconda3\Lib\importlib\metadata_init_.py:981, in distribution(distribution_name)
975 def distribution(distribution_name):
976 “”“Get the Distribution instance for the named package.
977
978 :param distribution_name: The name of the distribution package as a string.
979 :return: A Distribution instance (or subclass thereof).
980 “””
→ 981 return Distribution.from_name(distribution_name)

File ~\anaconda3\Lib\importlib\metadata_init_.py:565, in Distribution.from_name(cls, name)
563 return next(cls.discover(name=name))
564 except StopIteration:
→ 565 raise PackageNotFoundError(name)

PackageNotFoundError: No package metadata was found for py-smps

Here is the error I get when running 'pip install py-smps [–upgrade]

Note: you may need to restart the kernel to use updated packages.
ERROR: Exception:
Traceback (most recent call last):
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_vendor\packaging\requirements.py”, line 102, in init
req = REQUIREMENT.parseString(requirement_string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_vendor\pyparsing\core.py”, line 1141, in parse_string
raise exc.with_traceback(None)
pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found ‘[’ (at char 11), (line:1, col:12)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\cli\base_command.py”, line 169, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\cli\req_command.py”, line 248, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\commands\install.py”, line 342, in run
reqs = self.get_requirements(args, options, finder, session)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\cli\req_command.py”, line 411, in get_requirements
req_to_add = install_req_from_line(
^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\req\constructors.py”, line 395, in install_req_from_line
parts = parse_req_from_line(name, line_source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\req\constructors.py”, line 332, in parse_req_from_line
extras = convert_extras(extras_as_string)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\req\constructors.py”, line 57, in convert_extras
return get_requirement(“placeholder” + extras.lower()).extras
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_internal\utils\packaging.py”, line 45, in get_requirement
return Requirement(req_string)
^^^^^^^^^^^^^^^^^^^^^^^
File “C:\Users\silas\anaconda3\Lib\site-packages\pip_vendor\packaging\requirements.py”, line 104, in init
raise InvalidRequirement(
pip._vendor.packaging.requirements.InvalidRequirement: Parse error at “‘[–upgra’”: Expected string_end

When I try ‘pip install py-smps’ I get this error:

Collecting py-smps
Using cached py_smps-2.0.0-py3-none-any.whl (19 kB)
Requirement already satisfied: joblib<2.0.0,>=1.0.0 in c:\users\silas\anaconda3\lib\site-packages (from py-smps) (1.2.0)
Requirement already satisfied: numpy<2.0.0,>=1.19.1 in c:\users\silas\anaconda3\lib\site-packages (from py-smps) (1.24.3)
Requirement already satisfied: pandas<2.0.0,>=1.2.4 in c:\users\silas\anaconda3\lib\site-packages (from py-smps) (1.5.3)
Requirement already satisfied: requests<3.0.0,>=2.24.0 in c:\users\silas\anaconda3\lib\site-packages (from py-smps) (2.29.0)
Requirement already satisfied: scipy<2.0.0,>=1.5.2 in c:\users\silas\anaconda3\lib\site-packages (from py-smps) (1.10.1)
Collecting seaborn<0.11.0,>=0.10.1 (from py-smps)
Using cached seaborn-0.10.1-py3-none-any.whl (215 kB)
Collecting statsmodels<0.13.0,>=0.12.0 (from py-smps)
Using cached statsmodels-0.12.2.tar.gz (17.5 MB)
Installing build dependencies: started
Installing build dependencies: finished with status ‘error’
Note: you may need to restart the kernel to use updated packages.
error: subprocess-exited-with-error

pip subprocess to install build dependencies did not run successfully.
exit code: 1

[96 lines of output]
Ignoring numpy: markers ‘python_version == “3.6”’ don’t match your environment
Ignoring numpy: markers ‘python_version == “3.7”’ don’t match your environment
Collecting setuptools
Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
Collecting wheel
Using cached wheel-0.41.0-py3-none-any.whl (64 kB)
Collecting cython>=0.29.14
Using cached Cython-3.0.0-cp311-cp311-win_amd64.whl (2.8 MB)
Collecting numpy==1.17.5
Using cached numpy-1.17.5.zip (6.4 MB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status ‘done’
Collecting scipy>=1.2
Using cached scipy-1.11.1-cp311-cp311-win_amd64.whl (44.0 MB)
INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
Using cached scipy-1.10.1-cp311-cp311-win_amd64.whl (42.2 MB)
Using cached scipy-1.10.0-cp311-cp311-win_amd64.whl (42.2 MB)
Using cached scipy-1.9.3-cp311-cp311-win_amd64.whl (39.9 MB)
Using cached scipy-1.9.2-cp311-cp311-win_amd64.whl (39.9 MB)
Using cached scipy-1.9.1.tar.gz (42.0 MB)
Installing build dependencies: started
Installing build dependencies: finished with status ‘done’
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status ‘error’
error: subprocess-exited-with-error

Getting requirements to build wheel did not run successfully.
exit code: 1

[55 lines of output]
The Meson build system
Version: 0.62.2
Source dir: C:\Users\silas\AppData\Local\Temp\pip-install-49lc5qvd\scipy_9b15cafb5bcc4de985ebb2760fce70b9
Build dir: C:\Users\silas\AppData\Local\Temp\pip-install-49lc5qvd\scipy_9b15cafb5bcc4de985ebb2760fce70b9\.mesonpy-v8n1bfwn\build
Build type: native build
Project name: SciPy
Project version: 1.9.1
WARNING: Failed to activate VS environment: Could not find C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe

..\..\meson.build:1:0: ERROR: Unknown compiler(s): [['icl'], ['cl'], ['cc'], ['gcc'], ['clang'], ['clang-cl'], ['pgcc']]
The following exception(s) were encountered:
Running "icl " gave "[WinError 2] The system cannot find the file specified"
Running "cl /?" gave "[WinError 2] The system cannot find the file specified"
Running "cc --version" gave "[WinError 2] The system cannot find the file specified"
Running "gcc --version" gave "[WinError 2] The system cannot find the file specified"
Running "clang --version" gave "[WinError 2] The system cannot find the file specified"
Running "clang-cl /?" gave "[WinError 2] The system cannot find the file specified"
Running "pgcc --version" gave "[WinError 2] The system cannot find the file specified"

A full log can be found at C:\Users\silas\AppData\Local\Temp\pip-install-49lc5qvd\scipy_9b15cafb5bcc4de985ebb2760fce70b9\.mesonpy-v8n1bfwn\build\meson-logs\meson-log.txt
+ meson setup --native-file=C:\Users\silas\AppData\Local\Temp\pip-install-49lc5qvd\scipy_9b15cafb5bcc4de985ebb2760fce70b9\.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=C:\Users\silas\anaconda3 C:\Users\silas\AppData\Local\Temp\pip-install-49lc5qvd\scipy_9b15cafb5bcc4de985ebb2760fce70b9 C:\Users\silas\AppData\Local\Temp\pip-install-49lc5qvd\scipy_9b15cafb5bcc4de985ebb2760fce70b9\.mesonpy-v8n1bfwn\build
Traceback (most recent call last):
  File "C:\Users\silas\anaconda3\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
    main()
  File "C:\Users\silas\anaconda3\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\silas\anaconda3\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 969, in get_requires_for_build_wheel
    with _project(config_settings) as project:
  File "C:\Users\silas\anaconda3\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 948, in _project
    with Project.with_temp_working_dir(
  File "C:\Users\silas\anaconda3\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 777, in with_temp_working_dir
    yield cls(source_dir, tmpdir, build_dir)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 682, in __init__
    self._configure(reconfigure=bool(build_dir) and not native_file_mismatch)
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 713, in _configure
    self._meson(
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 696, in _meson
    return self._proc('meson', *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\silas\AppData\Local\Temp\pip-build-env-l2kns0lo\overlay\Lib\site-packages\mesonpy\__init__.py", line 691, in _proc
    subprocess.check_call(list(args))
  File "C:\Users\silas\anaconda3\Lib\subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['meson', 'setup', '--native-file=C:\\Users\\silas\\AppData\\Local\\Temp\\pip-install-49lc5qvd\\scipy_9b15cafb5bcc4de985ebb2760fce70b9\\.mesonpy-native-file.ini', '-Ddebug=false', '-Doptimization=2', '--prefix=C:\\Users\\silas\\anaconda3', 'C:\\Users\\silas\\AppData\\Local\\Temp\\pip-install-49lc5qvd\\scipy_9b15cafb5bcc4de985ebb2760fce70b9', 'C:\\Users\\silas\\AppData\\Local\\Temp\\pip-install-49lc5qvd\\scipy_9b15cafb5bcc4de985ebb2760fce70b9\\.mesonpy-v8n1bfwn\\build']' returned non-zero exit status 1.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error

Getting requirements to build wheel did not run successfully.
exit code: 1

See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

pip subprocess to install build dependencies did not run successfully.
exit code: 1

See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Thanks for taking a look, anything helps!
Silas

dmcc · August 3, 2023, 7:16pm

Thanks Silas! I was able to reproduce your issues on Anaconda’s hosted environment – I’m afraid I don’t know Conda well enough to know how to make it behave.

Would using a different Python environment work for you? I’ve been able to install and use py-smps in Google Colab (example notebook). This should still work for being able to render and see its visualizations.

Hope this helps,
David