We propose the Poisson neural networks (PNNs) to learn Poisson systems and trajectories of autonomous systems from data. Based on the Darboux-Lie theorem, the phase flow of a Poisson system can be written as the composition of (1) a coordinate transformation, (2) an extended symplectic map and (3) the inverse of the transformation. In this work, we extend this result to the unknotted trajectories of autonomous systems. We employ structured neural networks with physical priors to approximate the three aforementioned maps. We demonstrate through several simulations that PNNs are capable of handling very accurately several challenging tasks, including the motion of a particle in the electromagnetic potential, the nonlinear Schr{ö}dinger equation, and pixel observations of the two-body problem.

The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the naïve approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especially pre-convergence, and its relation to other pruning methods, such as gradient based pruning, are outstanding open questions in the field that are in need of being addressed. We make use of recent advances in dynamical systems theory, namely Koopman operator theory, to define a new class of theoretically motivated pruning algorithms. We show that these algorithms can be equivalent to magnitude and gradient based pruning, unifying these seemingly disparate methods, and that they can be used to shed light on magnitude pruning's performance during early training.

We study the meta-learning of numerical algorithms for scientific computing, which combines the mathematically driven, handcrafted design of general algorithm structure with a data-driven adaptation to specific classes of tasks. This represents a departure from the classical approaches in numerical analysis, which typically do not feature such learning-based adaptations. As a case study, we develop a machine learning approach that automatically learns effective solvers for initial value problems in the form of ordinary differential equations (ODEs), based on the Runge-Kutta (RK) integrator architecture. By combining neural network approximations and meta-learning, we show that we can obtain high-order integrators for targeted families of differential equations without the need for computing integrator coefficients by hand. Moreover, we demonstrate that in certain cases we can obtain superior performance to classical RK methods. This can be attributed to certain properties of the ODE families being identified and exploited by the approach. Overall, this work demonstrates an effective, learning-based approach to the design of algorithms for the numerical solution of differential equations, an approach that can be readily extended to other numerical tasks.

Scientists and engineers often create accurate, trustworthy, computational simulation schemes-but all too often these are too computationally expensive to execute over the time or spatial domain of interest. The equation-free approach is to marry such trusted simulations to a framework for numerical macroscale reduction-the patch dynamics scheme. This article extends the patch scheme to scenarios in which the trusted simulation resolves abrupt state changes on the microscale that appear as shocks on the macroscale. Accurate simulation for problems in these scenarios requires capturing the shock within a novel patch, and also modifying the patch coupling rules in the vicinity in order to maintain accuracy. With these two extensions to the patch scheme, straightforward arguments derive consistency conditions that match the usual order of accuracy for patch schemes. The new scheme is successfully tested to simulate a heterogeneous microscale partial differential equation. This technique will empower scientists and engineers to accurately and efficiently simulate, over large spatial domains, multiscale multiphysics systems that have rapid transition layers on the microscale.

A multiscale computational scheme is developed to use given small microscale simulations of complicated physical wave processes to empower macroscale system-level predictions. By coupling small patches of simulations over unsimulated space, large savings in computational time are realizable. Here, we generalize the patch scheme to the case of wave systems on staggered grids in two-dimensional (2D) space. Classic macroscale interpolation provides a generic coupling between patches that achieves consistency between the emergent macroscale simulation and the underlying microscale dynamics. Spectral analysis indicates that the resultant scheme empowers feasible computation of large macroscale simulations of wave systems even with complicated underlying physics. As an example of the scheme's application, we use it to simulate some simple scenarios of a given turbulent shallow water model.

In the present work we explore the application of a few root-finding methods to a series of prototypical examples. The methods we consider include: (a) the so-called continuous-time Nesterov (CTN) flow method; (b) a variant thereof referred to as the squared-operator method (SOM); and (c) the joint action of each of the above two methods with the so-called deflation method. More “traditional” methods such as Newton’s method (and its variant with deflation) are also brought to bear. Our toy examples start with a naive one degree-of-freedom (DOF) system to provide the lay of the land. Subsequently, we turn to a 2-DOF system that is motivated by the reduction of an infinite-dimensional, phase field crystal (PFC) model of soft matter crystallisation. Once the landscape of the 2-DOF system has been elucidated, we turn to the full PDE model and illustrate how the insights of the low-dimensional examples lead to novel solutions at the PDE level that are of relevance and interest to the full framework of soft matter crystallisation.

Despite great progress in simulating multiphysics problems using the numerical discretization of partial differential equations (PDEs), one still cannot seamlessly incorporate noisy data into existing algorithms, mesh generation remains complex, and high-dimensional problems governed by parameterized PDEs cannot be tackled. Moreover, solving inverse problems with hidden physics is often prohibitively expensive and requires different formulations and elaborate computer codes. Machine learning has emerged as a promising alternative, but training deep neural networks requires big data, not always available for scientific problems. Instead, such networks can be trained from additional information obtained by enforcing the physical laws (for example, at random points in the continuous space-time domain). Such physics-informed learning integrates (noisy) data and mathematical models, and implements them through neural networks or other kernel-based regression networks. Moreover, it may be possible to design specialized network architectures that automatically satisfy some of the physical invariants for better accuracy, faster training and improved generalization. Here, we review some of the prevailing trends in embedding physics into machine learning, present some of the current capabilities and limitations and discuss diverse applications of physics-informed learning both for forward and inverse problems, including discovering hidden physics and tackling high-dimensional problems.

Equation-free macroscale modelling is a systematic and rigorous computational methodology for efficiently predicting the dynamics of a microscale complex system at a desired macroscale system level. In this scheme, a given microscale model is computed in small patches spread across the space-time domain, with patch coupling conditions bridging the unsimulated space. For accurate predictions, care must be taken in designing the patch coupling conditions. Here we construct novel coupling conditions which preserve self-adjoint symmetry, thus guaranteeing that the macroscale model maintains some important conservation laws of the original microscale model. Consistency of the patch scheme’s macroscale dynamics with the original microscale model is proved for systems in 1D and 2D space, and these proofs immediately extend to higher dimensions. Expanding from a system with a single configuration to an ensemble of configurations establishes that the proven consistency also holds for cases where the microscale periodicity does not integrally fill the patches. This new self-adjoint patch scheme provides an efficient, flexible, and accurate computational homogenisation, as demonstrated here with canonical examples in 1D and 2D space based on heterogenous diffusion, and is applicable to a wide range of multiscale scenarios of interest to scientists and engineers.

Equations governing physico-chemical processes are usually known at microscopic spatial scales, yet one suspects that there exist equations, e.g., in the form of partial differential equations (PDEs), that can explain the system evolution at much coarser, meso-, or macroscopic length scales. Discovering those coarse-grained effective PDEs can lead to considerable savings in computation-intensive tasks like prediction or control. We propose a framework combining artificial neural networks with multiscale computation, in the form of equation-free numerics, for the efficient discovery of such macro-scale PDEs directly from microscopic simulations. Gathering sufficient microscopic data for training neural networks can be computationally prohibitive; equation-free numerics enable a more parsimonious collection of training data by only operating in a sparse subset of the space-time domain. We also propose using a data-driven approach, based on manifold learning (including one using the notion of unnormalized optimal transport of distributions and one based on moment-based description of the distributions), to identify macro-scale dependent variable(s) suitable for the data-driven discovery of said PDEs. This approach can corroborate physically motivated candidate variables or introduce new data-driven variables, in terms of which the coarse-grained effective PDE can be formulated. We illustrate our approach by extracting coarse-grained evolution equations from particle-based simulations with a priori unknown macro-scale variable(s) while significantly reducing the requisite data collection computational effort.

The study of nonlinear waves that collapse in finite time is a theme of universal interest, e.g., within optical, at., plasma physics, and nonlinear dynamics. Here we revisit the quintessential example of the nonlinear Schrödinger equation and systematically derive a normal form for the emergence of radially sym. blowup solutions from stationary ones. While this is an extensively studied problem, such a normal form, based on the methodol. of asymptotics beyond all algebraic orders, applies to both the dimension-dependent and power-law-dependent bifurcations previously studied. It yields excellent agreement with numerics in both leading and higher-order effects, it is applicable to both infinite and finite domains, and it is valid in both critical and supercritical regimes.

Simulations of large-scale gas-particle flows using coarse meshes and the filtered two-fluid model approach depend critically on the constitutive model that accounts for the effects of sub-grid scale inhomogeneous structures. In an earlier study (Jiang et al., 2019), we had demonstrated that an artificial neural network (ANN) model for drag correction developed from a small-scale systems did well in both a priori and a posteriori tests. In the present study, we first demonstrate through a cascading anal. that the extrapolation of the ANN model to large grid sizes works satisfactorily, and then performed fine-grid simulations for 20 addnl. combinations of gas and particle properties straddling the Geldart A-B transition. We identified the Reynolds number as a suitable addnl. marker to combine the results from all the different cases, and developed a general ANN model for drag correction that can be used for a range of gas and particle characteristics.

We present an approach, based on learning an intrinsic data manifold, for the initialization of the internal state values of long short-term memory (LSTM) recurrent neural networks, ensuring consistency with the initial observed input data. Exploiting the generalized synchronization concept, we argue that the converged, "mature" internal states constitute a function on this learned manifold. The dimension of this manifold then dictates the length of observed input time series data required for consistent initialization. We illustrate our approach through a partially observed chemical model system, where initializing the internal LSTM states in this fashion yields visibly improved performance. Finally, we show that learning this data manifold enables the transformation of partially observed dynamics into fully observed ones, facilitating alternative identification paths for nonlinear dynamical systems.

Dynamic control of engineered microbes using light via optogenetics has been demonstrated as an effective strategy for improving the yield of biofuels, chems., and other products. An advantage of using light to manipulate microbial metabolism is the relative simplicity of interfacing biol. and computer systems, thereby enabling in silico control of the microbe. Using this strategy for control and optimization of product yield requires an understanding of how the microbe responds in real-time to the light inputs. Toward this end, the authors present mechanistic models of a set of yeast optogenetic circuits. The authors show how these models can predict short- and long-time response to varying light inputs and how they are amenable to use with model predictive control (the industry standard among advanced control algorithms). These models reveal dynamics characterized by time-scale separation of different circuit components that affect the steady and transient levels of the protein under control of the circuit. Ultimately, this work will help enable real-time control and optimization tools for improving yield and consistency in the production of biofuels and chems. using microbial fermentations

Landscapes play an important role in many areas of biology, in which biological lives are deeply entangled. Here we discuss a form of landscape in evolutionary biology which takes into account (1) initial growth rates, (2) mutation rates, (3) resource consumption by organisms, and (4) cyclic changes in the resources with time. The long-term equilibrium number of surviving organisms as a function of these four parameters forms what we call a success landscape, a landscape we would claim is qualitatively different from fitness landscapes which commonly do not include mutations or resource consumption/changes in mapping genomes to the final number of survivors. Although our analysis is purely theoretical, we believe the results have possibly strong connections to how we might treat diseases such as cancer in the future with a deeper understanding of the interplay between resource degradation, mutation, and uncontrolled cell growth.

Large collections of coupled, heterogeneous agents can manifest complex dynamical behavior presenting difficulties for simulation and analysis. However, if the collective dynamics lie on a low-dimensional manifold, then the original agent-based model may be approximated with a simplified surrogate model on and near the low-dimensional space where the dynamics live. Analytically identifying such simplified models can be challenging or impossible, but here we present a data-driven coarse-graining methodology for discovering such reduced models. We consider two types of reduced models: globally based models that use global information and predict dynamics using information from the whole ensemble and locally based models that use local information, that is, information from just a subset of agents close (close in heterogeneity space, not physical space) to an agent, to predict the dynamics of an agent. For both approaches, we are able to learn laws governing the behavior of the reduced system on the low-dimensional manifold directly from time series of states from the agent-based system. These laws take the form of either a system of ordinary differential equations (ODEs), for the globally based approach, or a partial differential equation (PDE) in the locally based case. For each technique, we employ a specialized artificial neural network integrator that has been templated on an Euler time stepper (i.e., a ResNet) to learn the laws of the reduced model. As part of our methodology, we utilize the proper orthogonal decomposition (POD) to identify the low-dimensional space of the dynamics. Our globally based technique uses the resulting POD basis to define a set of coordinates for the agent states in this space and then seeks to learn the time evolution of these coordinates as a system of ODEs. For the locally based technique, we propose a methodology for learning a partial differential equation representation of the agents; the PDE law depends on the state variables and partial derivatives of the state variables with respect to model heterogeneities. We require that the state variables are smooth with respect to model heterogeneities, which permit us to cast the discrete agent-based problem as a continuous one in heterogeneity space. The agents in such a representation bear similarity to the discretization points used in typical finite element/volume methods. As an illustration of the efficacy of our techniques, we consider a simplified coupled neuron model for rhythmic oscillations in the pre-Bötzinger complex and demonstrate how our data-driven surrogate models are able to produce dynamics comparable to the dynamics of the full system. A nontrivial conclusion is that the dynamics can be equally well reproduced by an all-to-all coupled and by a locally coupled model of the same agents.

ZnO deposition in porous γ-Al2O3 via at. layer deposition (ALD) is the critical first step for the fabrication of zeolitic imidazolate framework membranes using the ligand-induced perm-selectivation process (Science, 361 (2018), 1008-1011). A detailed computational fluid dynamics (CFD) model of the ALD reactor is developed using a finite-volume-based code and validated. It accounts for the transport processes within the feeding system and reaction chamber. The simulated precursor spatiotemporal profiles assuming no ALD reaction were used as boundary conditions in modeling diethylzinc reaction/diffusion in porous γ-Al2O3, the predictions of which agreed with exptl. electron microscopy measurements. Further simulations confirmed that the present deposition flux is much less than the upper limit of flux, below which the decoupling of reactor/substrate is an accurate assumption. The modeling approach demonstrated here allows for the design of ALD processes for thin-film membrane formation including the synthesis of metal-organic framework membranes.

Data mining is routinely used to organize ensembles of short temporal observations so as to reconstruct useful, low-dimensional realizations of an underlying dynamical system. In this paper, we use manifold learning to organize unstructured ensembles of observations ("trials") of a system's response surface. We have no control over where every trial starts, and during each trial, operating conditions are varied by turning "agnostic" knobs, which change system parameters in a systematic, but unknown way. As one (or more) knobs "turn," we record (possibly partial) observations of the system response. We demonstrate how such partial and disorganized observation ensembles can be integrated into coherent response surfaces whose dimension and parametrization can be systematically recovered in a data-driven fashion. The approach can be justified through the Whitney and Takens embedding theorems, allowing reconstruction of manifolds/attractors through different types of observations. We demonstrate our approach by organizing unstructured observations of response surfaces, including the reconstruction of a cusp bifurcation surface for hydrogen combustion in a continuous stirred tank reactor. Finally, we demonstrate how this observation-based reconstruction naturally leads to informative transport maps between the input parameter space and output/state variable spaces.

Complex spatiotemporal dynamics of physicochemical processes are often modeled at a microscopic level (through, e.g., atomistic, agent-based, or lattice models) based on first principles. Some of these processes can also be successfully modeled at the macroscopic level using, e.g., partial differential equations (PDEs) describing the evolution of the right few macroscopic observables (e.g., concentration and momentum fields). Deriving good macroscopic descriptions (the so-called "closure problem") is often a time-consuming process requiring deep understanding/intuition about the system of interest. Recent developments in data science provide alternative ways to effectively extract/learn accurate macroscopic descriptions approximating the underlying microscopic observations. In this paper, we introduce a data-driven framework for the identification of unavailable coarse-scale PDEs from microscopic observations via machine-learning algorithms. Specifically, using Gaussian processes, artificial neural networks, and/or diffusion maps, the proposed framework uncovers the relation between the relevant macroscopic space fields and their time evolution (the right-hand side of the explicitly unavailable macroscopic PDE). Interestingly, several choices equally representative of the data can be discovered. The framework will be illustrated through the data-driven discovery of macroscopic, concentration-level PDEs resulting from a fine-scale, lattice Boltzmann level model of a reaction/transport process. Once the coarse evolution law is identified, it can be simulated to produce long-term macroscopic predictions. Different features (pros as well as cons) of alternative machine-learning algorithms for performing this task (Gaussian processes and artificial neural networks) are presented and discussed.

Developing accurate dynamical system models from phys. insights or data can be impeded when only partial observations of the system state are available. Here, we combine conservation laws used in physics and engineering with artificial neural networks to construct "gray-box" system models that make accurate predictions even with limited information. These models use a time delay embedding (cf., Takens embedding theorem) to reconstruct the effects of the intrinsic states, and they can be used for multiscale systems where macroscopic balance equations depend on unmeasured micro/meso-scale phenomena. By incorporating physics knowledge into the neural network architecture, we regularize variables and can train the model more accurately on smaller data sets than black-box neural network models. We present numerical examples from biotechnol., including a continuous bioreactor actuated using light through optogenetics (an emerging technol. in synthetic biol.) where the effect of unmeasured intracellular information is recovered from the histories of the measured macroscopic variables.

Model predictive control (MPC) is a de facto standard control algorithm across the process industries. There remain, however, applications where MPC is impractical because an optimization problem is solved at each time step. We present a link between explicit MPC formulations and manifold learning to enable facilitated prediction of the MPC policy. Our method uses a similarity measure informed by control policies and system state variables, to "learn" an intrinsic parametrization of the MPC controller using a diffusion maps algorithm, which will also discover a low-dimensional control law when it exists as a smooth, nonlinear combination of the state variables. We use function approximation algorithms to project points from state space to the intrinsic space, and from the intrinsic space to policy space. The approach is illustrated first by "learning" the intrinsic variables for MPC control of constrained linear systems, and then by designing controllers for an unstable nonlinear reactor.

We propose a local conformal autoencoder (LOCA) for standardized data coordinates. LOCA is a deep learning-based method for obtaining standardized data coordinates from scientific measurements. Data observations are modeled as samples from an unknown, nonlinear deformation of an underlying Riemannian manifold, which is parametrized by a few normalized, latent variables. We assume a repeated measurement sampling strategy, common in scientific measurements, and present a method for learning an embedding in Rd that is isometric to the latent variables of the manifold. The coordinates recovered by our method are invariant to diffeomorphisms of the manifold, making it possible to match between different instrumental observations of the same phenomenon. Our embedding is obtained using LOCA, which is an algorithm that learns to rectify deformations by using a local z-scoring procedure, while preserving relevant geometric information. We demonstrate the isometric embedding properties of LOCA in various model settings and observe that it exhibits promising interpolation and extrapolation capabilities, superior to the current state of the art. Finally, we demonstrate LOCA′s efficacy in single-site Wi-Fi localization data and for the reconstruction of three-dimensional curved surfaces from two-dimensional projections.

Systems of coupled dynamical units (e.g., oscillators or neurons) are known to exhibit complex, emergent behaviors that may be simplified through coarse-graining: a process in which one discovers coarse variables and derives equations for their evolution. Such coarse-graining procedures often require extensive experience and/or a deep understanding of the system dynamics. In this paper we present a systematic, data-driven approach to discovering "bespoke" coarse variables based on manifold learning algorithms. We illustrate this methodology with the classic Kuramoto phase oscillator model, and demonstrate how our manifold learning technique can successfully identify a coarse variable that is one-to-one with the established Kuramoto order parameter. We then introduce an extension of our coarse-graining methodology which enables us to learn evolution equations for the discovered coarse variables via an artificial neural network architecture templated on numerical time integrators (initial value solvers). This approach allows us to learn accurate approximations of time derivatives of state variables from sparse flow data, and hence discover useful approximate differential equation descriptions of their dynamic behavior. We demonstrate this capability by learning ODEs that agree with the known analytical expression for the Kuramoto order parameter dynamics at the continuum limit. We then show how this approach can also be used to learn the dynamics of coarse variables discovered through our manifold learning methodology. In both of these examples, we compare the results of our neural network based method to typical finite differences complemented with geometric harmonics. Finally, we present a series of computational examples illustrating how a variation of our manifold learning methodology can be used to discover sets of "effective" parameters, reduced parameter combinations, for multi-parameter models with complex coupling. We conclude with a discussion of possible extensions of this approach, including the possibility of obtaining data-driven effective partial differential equations for coarse-grained neuronal network behavior, as illustrated by the synchronization dynamics of Hodgkin-Huxley type neurons with a Chung-Lu network. Thus, we build an integrated suite of tools for obtaining data-driven coarse variables, data-driven effective parameters, and data-driven coarse-grained equations from detailed observations of networks of oscillators.

The high computational requirements of nonlinear model predictive control (NMPC) are a long-standing issue and, among other methods, learning the control policy with machine learning (ML) methods has been proposed in order to improve computational tractability. However, these methods typically do not explicitly consider constraint satisfaction. We propose two methods based on learning the optimal control policy by an artificial neural network (ANN) and using this for initialization to accelerate computations while meeting constraints and achieving good objective function value. In the first, the ANN prediction serves as the initial guess for the solution of the optimal control problem (OCP) solved in NMPC. In the second, the ANN prediction is improved by solving a single quadratic program (QP). We compare the performance of the two proposed strategies against two benchmarks representing the extreme cases of (i) solving the NMPC problem to convergence using the shift-initialization strategy and (ii) implementing the controls predicted by the ANN prediction without further correction to reduce the computational delay. We find that the proposed ANN initialization strategy mostly results in the same control policy as the shift-initialization strategy. The computational times are on average ∼ 45% longer but the maximum time is ∼ 42% smaller and the distribution is tighter, thus more predictable. The proposed QP-based method yields a good compromise between finding the optimal control policy and solution time. Closed-loop infeasibilities are negligible and the objective function is typically greatly improved as compared to benchmark (ii). The computational time required for the necessary second-order sensitivity integration is typically an order of magnitude smaller than for solving the NMPC problem to convergence.

The use of optogenetics in metabolic engineering for light-controlled microbial chem. production raises the prospect of utilizing control and optimization techniques routinely deployed in traditional chem. manufacturing However, such mechanisms require well-characterized, customizable tools that respond fast enough to be used as real-time inputs during fermentations Here, we present OptoINVRT7, a new rapid optogenetic inverter circuit to control gene expression in Saccharomyces cerevisiae. The circuit induces gene expression in only 0.6 h after switching cells from light to darkness, which is at least 6 times faster than previous OptoINVRT optogenetic circuits used for chem. production In addition, we introduce an engineered inducible GAL1 promoter (PGAL1-S), which is stronger than any constitutive or inducible promoter commonly used in yeast. Combining OptoINVRT7 with PGAL1-S achieves strong and light-tunable levels of gene expression with as much as 132.9 ± 22.6-fold induction in darkness. The high performance of this new optogenetic circuit in controlling metabolic enzymes boosts production of lactic acid and isobutanol by more than 50% and 15%, resp. The strength and controllability of OptoINVRT7 and PGAL1-S open the door to applying process control tools to engineered metabolisms to improve robustness and yields in microbial fermentations for chem. production

Concise, accurate descriptions of phys. systems through their conserved quantities abound in the natural sciences. In data science, however, current research often focuses on regression problems, without routinely incorpo- rating addnl. assumptions about the system that generated the data. Here, we propose to explore a particular type of underlying structure in the data: Hamiltonian systems, where an "energy" is conserved. Given a collection of observations of such a Hamiltonian system over time, we extract phase space coordinates and a Hamiltonian function of them that acts as the generator of the system dynamics. The approach employs an auto-encoder neural network component to estimate the transformation from observations to the phase space of a Hamiltonian system. An addi- tional neural network component is used to approx. the Hamiltonian function on this constructed space, and the two components are trained simultaneously. As an alternative approach, we also demonstrate the use of Gaussian processes for the estimation of such a Hamiltonian. After two illustrative examples, we extract an underlying phase space as well as the generating Hamiltonian from a collection of movies of a pendulum. The approach is fully data-driven, and does not assume a particular form of the Hamiltonian function.

Concise, accurate descriptions of physical systems through their conserved quantities abound in the natural sciences. In data science, however, current research often focuses on regression problems, without routinely incorporating additional assumptions about the system that generated the data. Here, we propose to explore a particular type of underlying structure in the data: Hamiltonian systems, where an "energy" is conserved. Given a collection of observations of such a Hamiltonian system over time, we extract phase space coordinates and a Hamiltonian function of them that acts as the generator of the system dynamics. The approach employs an autoencoder neural network component to estimate the transformation from observations to the phase space of a Hamiltonian system. An additional neural network component is used to approximate the Hamiltonian function on this constructed space, and the two components are trained jointly. As an alternative approach, we also demonstrate the use of Gaussian processes for the estimation of such a Hamiltonian. After two illustrative examples, we extract an underlying phase space as well as the generating Hamiltonian from a collection of movies of a pendulum. The approach is fully data-driven and does not assume a particular form of the Hamiltonian function.

The multiscale patch scheme is built from given small micro-scale simulations of complicated phys. processes to empower large macro-scale simulations. By coupling small patches of simulations over unsimulated spatial gaps, large savings in computational time are possible. Here we discuss generalising the patch scheme to the case of wave systems on staggered grids in 2D space. Classic macro-scale interpolation provides a generic coupling between patches that achieves arbitrarily high order consistency between the emergent macro-scale simulation and the underlying micro-scale dynamics. Eigen-anal. indicates that the resultant scheme empowers feasible computation of large macro-scale simulations of wave systems even with complicated underlying physics. As examples we use the scheme to simulate some wave scenarios via a turbulent shallow water model.

Large scale dynamical systems (e.g. many nonlinear coupled differential equations) can often be summarized in terms of only a few state variables (a few equations), a trait that reduces complexity and facilitates exploration of behavioral aspects of otherwise intractable models. High model dimensionality and complexity makes symbolic, pen-and-paper model reduction tedious and impractical, a difficulty addressed by recently developed frameworks that computerize reduction. Symbolic work has the benefit, however, of identifying both reduced state variables and parameter combinations that matter most (effective parameters, "inputs"); whereas current computational reduction schemes leave the parameter reduction aspect mostly unaddressed. As the interest in mapping out and optimizing complex input-output relations keeps growing, it becomes clear that combating the curse of dimensionality also requires efficient schemes for input space exploration and reduction. Here, we explore systematic, data-driven parameter reduction by means of effective parameter identification, starting from current nonlinear manifoldlearning techniques enabling state space reduction. Our approach aspires to extend the data-driven determination of effective state variables with the data-driven discovery of effective model parameters, and thus to accelerate the exploration of high-dimensional parameter spaces associated with complex models.

In statistical modelling with Gaussian process regression, it has been shown that combining (few) high-fidelity data with (many) low-fidelity data can enhance prediction accuracy, compared to prediction based on the few high-fidelity data only. Such information fusion techniques for multi-fidelity data commonly approach the high-fidelity model fh(t) as a function of two variables (t, s), and then use fl(t) as the s data. More generally, the high-fidelity model can be written as a function of several variables (t, s1, s2….); the low-fidelity model fl and, say, some of its derivatives can then be substituted for these variables. In this paper, we will explore mathematical algorithms for multi-fidelity information fusion that use such an approach towards improving the representation of the high-fidelity function with only a few training data points. Given that fh may not be a simple function-and sometimes not even a function-of fl, we demonstrate that using additional functions of t, such as derivatives or shifts of fl, can drastically improve the approximation of fh through Gaussian processes. We also point out a connection with 'embedology' techniques from topology and dynamical systems. Our illustrative examples range from instructive caricatures to computational biology models, such as Hodgkin-Huxley neural oscillations.

The ultimate goal of physics is finding a unique equation capable of describing the evolution of any observable quantity in a self-consistent way. Within the field of statistical physics, such an equation is known as the generalized Langevin equation (GLE). Nevertheless, the formal and exact GLE is not particularly useful, since it depends on the complete history of the observable at hand, and on hidden degrees of freedom typically inaccessible from a theor. point of view. In this work, we propose the use of deep neural networks as a new avenue for learning the intricacies of the unknowns mentioned above. By using machine learning to eliminate the unknowns from GLEs, our methodol. outperforms previous approaches (in terms of efficiency and robustness) where general fitting functions were postulated. Finally, our work is tested against several prototypical examples, from a colloidal systems and particle chains immersed in a thermal bath, to climatol. and financial models. In all cases, our methodol. exhibits an excellent agreement with the actual dynamics of the observables under consideration.

Nonlinear manifold learning algorithms, such as diffusion maps, have been fruitfully applied in recent years to the analysis of large and complex data sets. However, such algorithms still encounter challenges when faced with real data. One such challenge is the existence of “repeated eigendirections,” which obscures the detection of the true dimensionality of the underlying manifold and arises when several embedding coordinates parametrize the same direction in the intrinsic geometry of the data set. We propose an algorithm, based on local linear regression, to automatically detect coordinates corresponding to repeated eigendirections. We construct a more parsimonious embedding using only the eigenvectors corresponding to unique eigendirections, and we show that this reduced diffusion maps embedding induces a metric which is equivalent to the standard diffusion distance. We first demonstrate the utility and flexibility of our approach on synthetic data sets. We then apply our algorithm to data collected from a stochastic model of cellular chemotaxis, where our approach for factoring out repeated eigendirections allows us to detect changes in dynamical behavior and the underlying intrinsic system dimensionality directly from data.

Recent studies have demonstrated that enzyme-instructed self-assembly (EISA) in extra- or intracellular environments can serve as a multistep process for controlling cell fate. There is little knowledge, however, about the kinetics of EISA in the complex environments in or around cells. Here, we design and synthesize three dipeptidic precursors (LD-1-SO3, DL-1-SO3, DD-1-SO3), consisting of diphenylalanine (L-Phe-D-Phe, D-Phe-L-Phe, D-Phe-D-Phe, resp.) as the backbone, which are capped by 2-(naphthalen-2-yl)acetic acid at the N-terminal and by 2-(4-(2-aminoethoxy)-4-oxobutanamido)ethane-1-sulfonic acid at the C-terminal. On hydrolysis by carboxylesterases (CES), these precursors result in hydrogelators, which self-assemble in water at different rates. Whereas all three precursors selectively kill cancer cells, especially high-grade serous ovarian carcinoma cells, by undergoing intracellular EISA, DL-1-SO3 and DD-1-SO3 exhibit the lowest and the highest activities, resp., against the cancer cells. This trend inversely correlates with the rates of converting the precursors to the hydrogelators in phosphate-buffered saline. Because CES exists both extra- and intracellularly, we use kinetic modeling to analyze the kinetics of EISA inside cells and to calculate the cytotoxicity of each precursor for killing cancer cells. Our results indicate that (i) the stereochem. of the precursors affects the morphol. of the nanostructures formed by the hydrogelators, as well as the rate of enzymic conversion; (ii) decreased extracellular hydrolysis of precursors favors intracellular EISA inside the cells; (iii) the inherent features (e.g., self-assembling ability and morphol.) of the EISA mols. largely dictate the cytotoxicity of intracellular EISA. As the kinetic anal. of intracellular EISA, this work elucidates how the stereochem. modulates EISA in the complex extra- and/or intracellular environment for developing anticancer mol. processes. Moreover, it provides insights for understanding the kinetics and cytotoxicity of aggregates of aberrant proteins or peptides formed inside and outside cells.

When studying observations of chem. reaction dynamics, closed form equations based on a putative mechanism may not be available. Yet when sufficient data from exptl. observations can be obtained, even without knowing what exactly the phys. meaning of the parameter settings or recorded variables are, data-driven methods can be used to construct minimal (and in a sense, robust) realizations of the system. The approach attempts, in a sense, to circumvent phys. understanding, by building intrinsic "information geometries" of the observed data, and thus enabling prediction without phys./chem. knowledge. Here we use such an approach to obtain evolution equations for a data-driven realization of the original system - in effect, allowing prediction based on the informed interrogation of the agnostically organized observation database. We illustrate the approach on observations of (a) the normal form for the cusp singularity, (b) a cusp singularity for the nonisothermal CSTR, and (c) a random invertible transformation of the nonisothermal CSTR, showing that one can predict even when the observables are not "simply explainable" phys. quantities. We discuss current limitations and possible extensions of the procedure.

Data mining is routinely used to organize ensembles of short temporal observations so as to re-construct useful, low-dimensional realizations of the underlying dynamical systems. By analogy, we use data mining to organize ensembles of a different type of short observations to reconstruct useful realizations of bifurcation diagrams. Here the observations arise not through temporal variation, but rather through the variation of input parameters to the system: typical examples include short one-parameter steady state continuation runs, recording components of the steady state along the continuation path segment. We demonstrate how partial and disorganized "bifurcation observations" can be integrated in coherent bifurcation surfaces whose dimensionality and topol./parametrization can be systematically recovered in a data-driven fashion. The approach can be justified through the Whitney and Takens embedding theorems, allowing reconstruction of manifolds/attractors through observations. We discuss extensions to different types of bifurcation observations (not just one-parameter continuation), and show how certain choices of observables create analogies between the observation of bifurcation surfaces and the observation of vector fields. Finally, we demonstrate how this observation-based reconstruction naturally leads to the construction of transport maps between input parameter space and output/state variable spaces.

The macroscopic properties of mol. materials can be drastically influenced by their solid-state packing arrangements, of which there can be many (e.g., polymorphism). Strategies to controllably and predictively access select polymorphs are thus highly desired, but computationally predicting the conditions necessary to access a given polymorph is challenging with the current state of the art. Using derivatives of contorted hexabenzocoronene, cHBC, we employed data mining, rather than first-principles approaches, to find relationships between the crystallizing mol., postdeposition solvent-vapor annealing conditions that induce polymorphic transformation, and the resulting polymorphs. This anal. yields a correlative function that can be used to successfully predict the appearance of either one of two polymorphs in thin films of cHBC derivatives Within the postdeposition processing phase space of cHBC derivatives, we have demonstrated an approach to generate guidelines to select crystallization conditions to bias polymorph access. We believe this approach can be applied more broadly to accelerate the predictions of processing conditions to access desired mol. polymorphs, making progress toward one of the grand challenges identified by the Materials Genome Initiative.

Manifold-learning techniques are routinely used in mining complex spatiotemporal data to extract useful, parsimonious data representations/parametrizations; these are, in turn, useful in nonlinear model identification tasks. We focus here on the case of time series data that can ultimately be modelled as a spatially distributed system (e.g. a partial differential equation, PDE), but where we do not know the space in which this PDE should be formulated. Hence, even the spatial coordinates for the distributed system themselves need to be identified - to "emerge from"-the data mining process. We will first validate this "emergent space" reconstruction for time series sampled without space labels in known PDEs; this brings up the issue of observability of physical space from temporal observation data, and the transition from spatially resolved to lumped (order-parameter-based) representations by tuning the scale of the data mining kernels. We will then present actual emergent space "discovery" illustrations. Our illustrative examples include chimera states (states of coexisting coherent and incoherent dynamics), and chaotic as well as quasiperiodic spatiotemporal dynamics, arising in partial differential equations and/or in heterogeneous networks. We also discuss how data-driven "spatial" coordinates can be extracted in ways invariant to the nature of the measuring instrument. Such gauge-invariant data mining can go beyond the fusion of heterogeneous observations of the same system, to the possible matching of apparently different systems. For an older version of this article, including other examples, see https://arxiv.org/abs/1708.05406.

Finding accurate reduced descriptions for large, complex, dynamically evolving networks is a crucial enabler to their simulation, analysis, and ultimately design. Here, we propose and illustrate a systematic and powerful approach to obtaining good collective coarse-grained observables-variables successfully summarizing the detailed state of such networks. Finding such variables can naturally lead to successful reduced dynamic models for the networks. The main premise enabling our approach is the assumption that the behavior of a node in the network depends (after a short initial transient) on the node identity: a set of descriptors that quantify the node properties, whether intrinsic (e.g., parameters in the node evolution equations) or structural (imparted to the node by its connectivity in the particular network structure). The approach creates a natural link with modeling and "computational enabling technology" developed in the context of Uncertainty Quantification. In our case, however, we will not focus on ensembles of different realizations of a problem, each with parameters randomly selected from a distribution. We will instead study many coupled heterogeneous units, each characterized by randomly assigned (heterogeneous) parameter value(s). One could then coin the term Heterogeneity Quantification for this approach, which we illustrate through a model dynamic network consisting of coupled oscillators with one intrinsic heterogeneity (oscillator individual frequency) and one structural heterogeneity (oscillator degree in the undirected network). The computational implementation of the approach, its shortcomings and possible extensions are also discussed.

We describe and implement iMapD, a computer-assisted approach for accelerating the exploration of uncharted effective Free Energy Surfaces (FES), and more genera ally for the extraction of coarse-grained, macroscopic information from atomistic or stochastic (here Mol. Dynamics, MD) simulations. The approach functionally links the MD simulator with nonlinear manifold learning techniques. The added value comes from biasing the simulator towards new, unexplored phase space regions by exploiting the smoothness of the (gradually, as the exploration progresses) revealed intrinsic low-dimensional geometry of the FES.

We describe and implement a computer-assisted approach for accelerating the exploration of uncharted effective free-energy surfaces (FESs). More generally, the aim is the extraction of coarse-grained, macroscopic information from stochastic or atomistic simulations, such as mol. dynamics (MD). The approach functionally links the MD simulator with nonlinear manifold learning techniques. The added value comes from biasing the simulator toward unexplored phase-space regions by exploiting the smoothness of the gradually revealed intrinsic low-dimensional geometry of the FES.

In recent work, we have illustrated the construction of an exploration geometry on free energy surfaces: the adaptive computer-assisted discovery of an approx. low-dimensional manifold on which the effective dynamics of the system evolves. Constructing such an exploration geometry involves geometry-biased sampling (through both appropriately-initialized unbiased mol. dynamics and through restraining potentials) and, machine learning techniques to organize the intrinsic geometry of the data resulting from the sampling (in particular, diffusion maps, possibly enhanced through the appropriate Mahalanobis-type metric). In this contribution, we detail a method for exploring the conformational space of a stochastic gradient system whose effective free energy surface depends on a smaller number of degrees of freedom than the dimension of the phase space. Our approach comprises two steps. First, we study the local geometry of the free energy landscape using diffusion maps on samples computed through stochastic dynamics. This allows us to automatically identify the relevant coarse variables. Next, we use the information garnered in the previous step to construct a new set of initial conditions for subsequent trajectories. These initial conditions are computed so as to explore the accessible conformational space more efficiently than by continuing the previous, unbiased simulations. We showcase this method on a representative test system.

We propose to compute approximations to invariant sets in dynamical systems by minimizing an appropriate distance between a suitably selected finite set of points and its image under the dynamics. We demonstrate, through computational experiments, that this approach can successfully converge to approximations of (maximal) invariant sets of arbitrary topol., dimension, and stability, such as, e.g., saddle type invariant sets with complicated dynamics. We further propose to extend this approach by adding a Lennard-Jones type potential term to the objective function, which yields more evenly distributed approximating finite point sets, and illustrate the procedure through corresponding numerical experiments (c) 2017 American Institute of Physics.

We discuss the interplay between manifold-learning tech- niques, which can extract intrinsic order from observations of complex dynamics, and systems modeling considerations. Tuning the scale of the data-mining kernels can guide the construction of dynamic models at different levels of coarse-graining. In particular, we focus on the observability of phys. space from temporal observation data and the transition from spatially resolved to lumped (order- parameter-based) representations, embedded in what we call an "Equal Space". Data-driven coordinates can be extracted in ways invariant to the nature of the measuring instrument. Such gauge-invariant data mining can then go beyond the fusion of heterogeneous observations of the same system, to the possible matching of apparently different systems. These techniques can enhance the scope and applicability of established tools in the anal. of dynamical systems, such as the Takens delay embedding. Our illustrative examples include chimera states (states of co-existing coherent and incoherent dynamics), and chaotic as well as quasiperiodic spatiotemporal dynamics, arising in partial differential equations and/or in heterogeneous networks.

Numerical approximation methods for the Koopman operator have advanced considerably in the last few years. In particular, data-driven approaches such as dynamic mode decomposition (DMD)51 and its generalization, the extended-DMD (EDMD), are becoming increasingly popular in practical applications. The EDMD improves upon the classical DMD by the inclusion of a flexible choice of dictionary of observables which spans a finite dimensional subspace on which the Koopman operator can be approximated. This enhances the accuracy of the solution reconstruction and broadens the applicability of the Koopman formalism. Although the convergence of the EDMD has been established, applying the method in practice requires a careful choice of the observables to improve convergence with just a finite number of terms. This is especially difficult for high dimensional and highly nonlinear systems. In this paper, we employ ideas from machine learning to improve upon the EDMD method. We develop an iterative approximation algorithm which couples the EDMD with a trainable dictionary represented by an artificial neural network. Using the Duffing oscillator and the Kuramoto Sivashinsky partical differential equation as examples, we show that our algorithm can effectively and efficiently adapt the trainable dictionary to the problem at hand to achieve good reconstruction accuracy without the need to choose a fixed dictionary a priori. Furthermore, to obtain a given accuracy, we require fewer dictionary terms than EDMD with fixed dictionaries. This alleviates an important shortcoming of the EDMD algorithm and enhances the applicability of the Koopman framework to practical problems.

The early Drosophila embryo provides unique opportunities for quant. studies of ERK signaling. This system is characterized by simple anatomy, the ease of obtaining large numbers of staged embryos, and the availability of powerful tools for genetic manipulation of the ERK pathway. Here, we describe how these exptl. advantages can be combined with recently developed microfluidic devices for high throughput imaging of ERK activation dynamics. We focus on the stage during the third hour of development, when ERK activation is essential for patterning of the future nerve cord. Our approach starts with an ensemble of fixed embryos stained with an antibody that recognizes the active, dually phosphorylated form of ERK. Each embryo in this ensemble provides a snapshot of the spatial and temporal pattern of ERK activation during development. We then quant. estimate the ages of fixed embryos using a model that links their morphol. and developmental time. This model is learned based on live imaging of cellularization and gastrulation, two highly stereotyped morphogenetic processes at this stage of embryogenesis. Applying this approach, we can characterize ERK signaling at high spatial and temporal resolution Our methodol. can be readily extended to studies of ERK regulation and function in multiple mutant backgrounds, providing a versatile assay for quant. studies of developmental ERK signaling.

We have constructed a microfabricated circular corral for bacteria made of rings of concentric funnels which channel motile bacteria outwards via non-hydrodynamic interactions with the funnel walls. Initially bacteria do move rapidly outwards to the periphery of the corral. At the edge, nano-slits allow for the transport of nutrients into the device while keeping the bacteria from escaping. After a period of time in which the bacteria increase their cell d. in this perimeter region, they are then able to defeat the phys. constrains of the funnels by launching back-propagating collective waves. We present the basic data and some nonlinear modeling which can explain how bacterial population waves propagate through a phys. funnel, and discuss possible biol. implications.

Collective cell migration underlies many biol. processes, including embryonic development, wound healing, and cancer progression. In the embryo, cells have been observed to move collectively in vortices using a mode of collective migration known as coherent angular motion (CAM). Here, to determine how CAM arises within a population and changes over time, we studied the motion of mammary epithelial cells within engineered monolayers, in which the cells moved collectively about a central axis in the tissue. Using quant. image anal., we found that CAM was significantly reduced when mitosis is suppressed. Particle-based simulations recreated the observed trends, suggesting that cell divisions drive the robust emergence of CAM and facilitate switches in the direction of collective rotation. Our simulations predicted that the location of a dividing cell, rather than the orientation of the division axis, facilitates the onset of this motion. These predictions agreed with exptl. observations, thereby providing, to our knowledge, new insight into how cell divisions influence CAM within a tissue. Overall, these findings highlighted the dynamic nature of CAM and suggested that regulating cell division is crucial for tuning emergent collective migratory behaviors, such as vortical motions observed in vivo.

Dynamical processes in biol. are studied using an ever-increasing number of techniques, each of which brings out unique features of the system. One of the current challenges is to develop systematic approaches for fusing heterogeneous datasets into an integrated view of multivariable dynamics. We demonstrate that heterogeneous data fusion can be successfully implemented within a semi-supervised learning framework that exploits the intrinsic geometry of high-dimensional datasets. We illustrate our approach using a dataset from studies of pattern formation in Drosophila. The result is a continuous trajectory that reveals the joint dynamics of gene expression, subcellular protein localization, protein phosphorylation, and tissue morphogenesis. Our approach can be readily adapted to other imaging modalities and forms a starting point for further steps of data analytics and modeling of biol. dynamics.

The photosensitive Belousov-Zhabotinsky (pBZ) reaction has been used extensively to study the properties of chem. oscillators. In particular, recent experiments revealed the existence of complex spatiotemporal dynamics for systems consisting of coupled micelles (V < 10-21 L) or droplets (V ≈ [10-8-10-11] L) in which the pBZ reaction takes place. These results have been mostly understood in terms of reaction-diffusion models. However, in view of the small size of the droplets and micelles, large fluctuations of concentrations are to be expected. In this work, the role is investigated of fluctuations on the dynamics of a single droplet with stochastic simulations of an extension of the Field-Koros-Noyes (FKN) model taking into account the photosensitivity. The birhythmicity and chaotic behaviors predicted by the FKN model in the absence of fluctuations become transient or intermittent regimes whose lifetime decreases with the size of the droplet. Simple oscillations are more robust and can be observed even in small systems (V > 10-12 L), which justifies the use of deterministic models in microfluidic systems of coupled oscillators. The simulations also reveal that fluctuations strongly affect the efficiency of inhibition by light, which is often used to control the kinetics of these systems: oscillations are found for parameter values for which they are supposed to be quenched according to deterministic predictions.

The discovery of phys. laws consistent with empirical observations is at the heart of (applied) science and engineering. These laws typically take the form of nonlinear differential equations depending on parameters; dynamical systems theory provides, through the appropriate normal forms, an "intrinsic" prototypical characterization of the types of dynamical regimes accessible to a given model. Using an implementation of data-informed geometry learning, we directly reconstruct the relevant "normal forms": a quant. mapping from empirical observations to prototypical realizations of the underlying dynamics. Interestingly, the state variables and the parameters of these realizations are inferred from the empirical observations; without prior knowledge or understanding, they parametrize the dynamics intrinsically without explicit reference to fundamental phys. quantities.

In order to illustrate the adaptation of traditional contin- uum numerical techniques to the study of complex network systems, we use the equation-free framework to analyze a dynamically evolv- ing multigraph. This approach is based on coupling short intervals of direct dynamic network simulation with appropriately-defined lifting and restriction operators, mapping the detailed network description to suitable macroscopic (coarse-grained) variables and back. This enables the acceleration of direct simulations through Coarse Projective Inte- gration (CPI), as well as the identification of coarse stationary states via a Newton-GMRES method. We also demonstrate the use of data- mining, both linear (principal component anal., PCA) and nonlin- ear (diffusion maps, DMAPS) to determine good macroscopic variables (observables) through which one can coarse-grain the model. These re- sults suggest methods for decreasing simulation times of dynamic real- world systems such as epidemiol. network models. Addnl., the data-mining techniques could be applied to a diverse class of prob- lems to search for a succint, low-dimensional description of the system in a small number of variables.