GitHub

Known issues

NetCDF: Not a valid data type or _FillValue type mismatch

Trying to define the _FillValue, produces the following error:

ERROR: LoadError: NCDatasets.NetCDFError(-45, "NetCDF: Not a valid data type or _FillValue type mismatch")

The error could be generated by a code like this:

using NCDatasets, DataStructures
# ...
tempvar = defVar(ds,"temp",Float32,("lonc","latc","time"), attrib = OrderedDict(
    "_FillValue" => -9999.))

or

using NCDatasets
# ...
tempvar = defVar(ds,"temp",Float32,("lonc","latc","time"), fillvalue = -9999.)

In fact, _FillValue must have the same data type as the corresponding variable. In the case above, tempvar is a 32-bit float and the number -9999. is a 64-bit float (aka double, which is the default floating point type in Julia). It is sufficient to convert the value -9999. to a 32-bit float -9999.f0 (or Float32(-9999.)).

Defining the attributes _FillValue, add_offset, scale_factor, units and calendar

An error like Cannotconvertan object of type Missing (or similar) is generated by code like this:

v = defVar(ds,"var_with_all_missing_data",Float32,("lon",))
v.attrib["_FillValue"] = fv
v[1] = missing

This produces the following error:

ERROR: LoadError: MethodError: Cannot `convert` an object of type Missing to an object of type Float32
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at number.jl:7
  convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250
  ...
Stacktrace:
 [1] fill!(::SubArray{Float32,1,NCDatasets.CFVariable{Float32,1,NCDatasets.Variable{Float32,1},NCDatasets.Attributes},Tuple{UnitRange{Int64}},false}, ::Missing) at ./multidimensional.jl:865
 [2] copyto! at ./broadcast.jl:871 [inlined]
 [3] materialize!(::SubArray{Float32,1,NCDatasets.CFVariable{Float32,1,NCDatasets.Variable{Float32,1},NCDatasets.Attributes},Tuple{UnitRange{Int64}},false}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Base.RefValue{Missing}}}) at ./broadcast.jl:822

One should use define the _FillValue (and similar attributes like add_offset, scale_factor, units and calendar affecting the type of the data) in the call of defVar:

using DataStructures
v = defVar(ds,"var_with_all_missing_data",Int32,("lon",), fillvalue = fv, attrib = OrderedDict(
    "scale_factor" => 0.1,
    "add_offset" => 0.1
    ))

This change was introduced in NCDatasets version 0.10

Multiple versions of HDF5 or NetCDF libraries

Having outdated versions of HDF5 or NetCDF libraries installed can be an issue on Windows if they are included in the system PATH environment variable. It is advised to adapt the system PATH to remove the locations containing these libraries.

Using a custom NetCDF library

The NetCDF library libnetcdf.so is installed as an artifact via the package NetCDF_jll. You can override which libnetcdf.so gets loaded through the Preferences package, as follows:

# install these packages if necessary
using Preferences, NetCDF_jll

set_preferences!(NetCDF_jll, "libnetcdf_path" => "/path/to/libnetcdf.so.xyz")

where /path/to/libnetcdf.so.xyz is the full path to the NetCDF library. This will create a LocalPreferences.toml file in your top-level project with the following content:

[NetCDF_jll]
libnetcdf_path = "/path/to/libnetcdf.so.xyz"

However, the dependencies of the library version libnetcdf.so.xyz (in particular libcurl.so and libmbedtls.so) should be compatible with the dependencies of julia (in the folder .../julia-x.y.z/lib/julia). On Linux, you can list the library dependencies with the shell command ldd, for example:

ldd /path/to/libnetcdf.so.xyz

OPeNDAP on Windows fails with Assertion failed: ocpanic

On windows, NetCDF 4.7.4 can fail with this error:

Assertion failed: ocpanic(("state->auth.curlflags.cookiejar != NULL")), file ocinternal.c, line 566

when accessing OPeNDAP URLs, like these:

using NCDatasets
ds = NCDataset("https://thredds.jpl.nasa.gov/thredds/dodsC/ncml_aggregation/OceanTemperature/modis/terra/11um/4km/aggregate__MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_DAYTIME_V2019.0.ncml#fillmismatch")

See also the issue report: https://github.com/Unidata/netcdf-c/issues/2380. The work-around is to create a .dodsrc in the current working directory with the content:

HTTP.COOKIEJAR=C:\Users\USERNAME\AppData\Local\Temp\

where USERNAME is your username. The directory should exist and be writable by the user. You can run pwd() to determine the current working directory. Note that the initial current working directory can be different depending you how you start julia (from the command line or from jupyter notebook for example). Julia need to be restarted after this file is placed in the your working directory.

version `CURL_4' not found

Julia and NetCDF_jll have several common dependencies (curl, MbedTLS, zlib). Non-official julia builds will work only if they use exactly the same library version as those used to compile NetCDF. This is unlikely to be the case in general and outside of our control. Therefore non-official julia builds are not supported. Official julia builds are available at https://julialang.org/downloads/.

Using for example the julia package from on Arch Linux, leads to this error:

julia> using NCDatasets
ERROR: LoadError: InitError: could not load library "/root/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so"
/usr/lib/julia/libcurl.so: version `CURL_4' not found (required by /root/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)
Stacktrace:
 [1] macro expansion
   @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]
 [2] __init__()
   @ NetCDF_jll ~/.julia/packages/NetCDF_jll/BYHmI/src/wrappers/x86_64-linux-gnu.jl:12
 [3] top-level scope
   @ stdin:1
during initialization of module NetCDF_jll

You will likely have similar issues with julia installed from other package managers (like Debian/Ubuntu apt, Homebrew...). The only supported solution is to install the offical julia builds.

Even when the official build of julia, this error can occur on Linux if an incompatible library is loaded when the user set LD_LIBRARY_PATH and LD_PRELOAD:

ERROR: LoadError: InitError: could not load library "/home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so"
/usr/lib/x86_64-linux-gnu/libcurl.so: version `CURL_4' not found (required by /home/user/.julia/artifacts/461703969206dd426cc6b4d99f69f6ffab2a9779/lib/libnetcdf.so)

Please make sure that your LD_LIBRARY_PATH and LD_PRELOAD are empty or verify that the all loaded libraries are binary compatible. You can check these environement variables by running the following commands in a terminal:

echo $LD_PRELOAD
echo $LD_LIBRARY_PATH

If you must set $LD_LIBRARY_PATH for some application, consider to use a wrapper script for this application or recompiling the application with the -rpath linker option rather than setting this variable globally.

Corner cases

  • An attribute representing a vector with a single value (e.g. [1]) will be read back as a scalar (1) (same behavior in python netCDF4 1.3.1).

  • NetCDF and Julia distinguishes between a vector of chars and a string, but both are returned as string for ease of use, in particular an attribute representing a vector of chars ['u','n','i','t','s'] will be read back as the string "units".

  • While reading a NetCDF time variable, the dates are converted using the Julia's DateTime (based on the proleptic Gregorian calendar following the ISO 8601 standard) when possible. When data is written to a NetCDF file (without specifying the calendar), the dates are saved using the default calendar of the NetCDF CF convention (the mixed Julian/Gregorian calendar, called "standard") when possible. It is recommended that the time origin specified by the units is after 15 October 1582 in which case the mixed Julian/Gregorian calendar is identical to the proleptic Gregorian calendar.