Fortran-Python Interface

Disclaimers: This question is reposted from SO upon SO user's suggestion to put it here since there is no specific code in question. This question is a subset of my larger theme of Fortran modernization. There are useful versions of this question asked already (1, 2, 3) and blog posts, and although helpful, I am curious of what is best practice now (some of those posts are 5-10 years old) and the context of my situation, which is the reason for asking it in a similar way. Background Our current code base is largely Fortran (400K LOC, most F90/some F77/maybe some newer versions), dispersed amongst some 10 separate dev teams (organizationally) in (somewhat) standalone modules. We are exploring the best practice idea of taking one of those modules and writing it in a faster-to-develop language (Python), and wrapping and extending bottlenecks in Fortran. However, since the module is currently completely in Fortran and interacts with other Fortran modules and a main Fortran loop, we need to keep the appearance or interface of the module to main Fortran loop the same. So the incremental change approach for this Fortran module – and other similarly architected Fortran modules – is to keep the interfaces the same and change the internals. In other words, the shell or signature of the Fortran subroutines or functions would be unchanged, but the functionality inside the function would be changed. This functionality inside the function is what we want to rewrite in Python. From discussions online - and my intuition, whatever that’s worth – it seems unadvised to embed Python like this, and instead do the reverse: extend Python with Fortran (for bottlenecks, as needed). However, this embedding Python approach seems the only logical, smallest, atomic step forward to isolate changes to the larger Fortran system. It appears that there are some options for this embedding Python problem. The most future-proof and stable way is via ISO_C_BINDING and using the Python C API (in theory…). There are also stable solutions via Cython and maybe ctypes that are reliable and well maintained. There are more dependency-heavy approaches like cffi and forpy, that introduce complexity for the benefit of not writing the additional C interface code. We are also aware of simple system calls of Python scripts from Fortran, but these seem too disk write/read heavy; we would really like to do minimal disk reads/writes and keep the array passing in memory (hopefully just pointer locations passed, not array copies). Another constraint is that this Fortran-Python interface should be able to handle most common datatypes and structures, including multi-dimensional arrays, or at least have a way to decompose complex custom datatypes to some collection of simpler datatypes like strings and arrays. Question: Given the above description, do you have any advice on what are the best practice ways that are currently available to call Python from Fortran, with minimal writing of additional C code (so a package like cffi would be preferred over Python C API approach)? using 2003 or 2008 Fortran versions; not sure if all 2018 features are implemented in our Intel Fortran compiler. Edit (2020-04-16): To specify this question further, I am rephrasing the question to be specific to writing as little C overhead code as possible.

Mar 14, 2025 - 21:51

Disclaimers:

This question is reposted from SO upon SO user's suggestion to put it here since there is no specific code in question.
This question is a subset of my larger theme of Fortran modernization.
There are useful versions of this question asked already (1, 2, 3) and blog posts, and although helpful, I am curious of what is best practice now (some of those posts are 5-10 years old) and the context of my situation, which is the reason for asking it in a similar way.

Background

Our current code base is largely Fortran (400K LOC, most F90/some F77/maybe some newer versions), dispersed amongst some 10 separate dev teams (organizationally) in (somewhat) standalone modules.

We are exploring the best practice idea of taking one of those modules and writing it in a faster-to-develop language (Python), and wrapping and extending bottlenecks in Fortran.

However, since the module is currently completely in Fortran and interacts with other Fortran modules and a main Fortran loop, we need to keep the appearance or interface of the module to main Fortran loop the same.

So the incremental change approach for this Fortran module – and other similarly architected Fortran modules – is to keep the interfaces the same and change the internals.

In other words, the shell or signature of the Fortran subroutines or functions would be unchanged, but the functionality inside the function would be changed.

This functionality inside the function is what we want to rewrite in Python.

From discussions online - and my intuition, whatever that’s worth – it seems unadvised to embed Python like this, and instead do the reverse: extend Python with Fortran (for bottlenecks, as needed).

However, this embedding Python approach seems the only logical, smallest, atomic step forward to isolate changes to the larger Fortran system.

It appears that there are some options for this embedding Python problem.

The most future-proof and stable way is via ISO_C_BINDING and using the Python C API (in theory…).

There are also stable solutions via Cython and maybe ctypes that are reliable and well maintained.

There are more dependency-heavy approaches like cffi and forpy, that introduce complexity for the benefit of not writing the additional C interface code.

We are also aware of simple system calls of Python scripts from Fortran, but these seem too disk write/read heavy; we would really like to do minimal disk reads/writes and keep the array passing in memory (hopefully just pointer locations passed, not array copies).

Another constraint is that this Fortran-Python interface should be able to handle most common datatypes and structures, including multi-dimensional arrays, or at least have a way to decompose complex custom datatypes to some collection of simpler datatypes like strings and arrays.

Question:

Given the above description, do you have any advice on what are the best practice ways that are currently available to call Python from Fortran, with minimal writing of additional C code (so a package like cffi would be preferred over Python C API approach)?

using 2003 or 2008 Fortran versions; not sure if all 2018 features are implemented in our Intel Fortran compiler.

Edit (2020-04-16):

To specify this question further, I am rephrasing the question to be specific to writing as little C overhead code as possible.