OpenEndpoints is software written in Java. The code is open source and licensed under Apache License with Commons Clause - see https://openendpoints.io/license.

For the deployment we recommend using Docker. A public Docker container is available at:

public.ecr.aws/x1t6d0t7/endpoints-he

Using Saxon-PE

By default the public Docker container will install the free XSLT processor Saxon-HE. This is sufficient for most purposes.

In order to deploy OpenEndpoints with the commercial version Saxon-PE you are required to buy a license for Saxon-PE (Professional Edition) and create a Docker container using that edition. The license isn't expensive and it's absolutely worth the money.

Here are the steps you need to take:

1. Purchase a Saxon-PE license at https://www.saxonica.com/shop/shop.html. You will get two files: the JAR file containing the Saxon-PE code, and also a Saxon-PE license file.

2. Install Java 21 and Maven on your computer if you have not already done so.

3. Check out the OpenEndpoints Git repo to your computer if you have not already done so.

4. Execute the following command to install the Saxon-PE JAR that you have purchased into your local Maven repository on your computer: mvn install:install-file -Dfile=<path to your Saxon-PE file> -DgroupId=net.sf.saxon -DartifactId=Saxon-PE -Dversion=9.7.0.18 -Dpackaging=jar replacing your path to your downloaded file as appropriate. Keep the -Dversion the same, no matter what version you've actually downloaded.

5. Copy the Saxon-PE license file to your Git checkout, placing it in the path saxon-pe/saxon-license.lic.

6. Execute the following command to build OpenEndpoints with Saxon-PE: mvn -DSaxon=PE clean package

7. Build the Docker image using a command like docker build -t endpoints-pe .

8. Push the Docker image to your Docker repository. Note that the terms of the Saxon-PE license do not allow you to make this Docker image public in any way.

The application can be deployed in two different ways.

In both scenarios

• Docker is used to deploy the application.

• A PostgreSQL database is required. No schema is needed, the application creates that itself the first time it runs.

• We recommend using a cloud with managed Docker services, e.g. ECS on AWS, or Kubernetes.

Multiple Application

This is the default option. You deploy the standard Docker image, you don't need to build your own.

The Service Portal is part of the installation:

• Applications are published from Git using the Service Portal.

• Application files are not stored inside the Docker image.

Single Application

This is a special option.

With this deployment mode the application directory is stored directly inside the Docker image. You build your own Docker image with your own Application files being part of that container.

Hence, only a single application will be available with this installation.

A PostgreSQL database is required prior to installing OpenEndpoints.

Database schema creation is not necessary; the software does that on startup.

The following data is saved in it:

• Request log: This saves metadata for each call, but not the data explicitly transmitted when the call is made. An exception is the debug mode, in which, as an exception, the data transferred when the call is made can also be saved.

• Incremental IDs: If the feature is used to provide incremental IDs, the last ID used is saved in the database.

• Forward-to-Endpoint-URL: OpenEndpoints can generate short URLs, which, when called, cause a request to a previously-specified endpoint including previously-saved request parameters.

• Service Portal: Application Data and Logins.

An application comprises of multiple endpoints. Each endpoint has a name which is used in the URL. Which endpoint is requested is specified when the user calls the application.

Send Request to Endpoint

OpenEndpoints knows 2 types of data sources that can serve as input for the content transformation:

• data that is transferred with the request, and

• data that is loaded from other sources when the endpoint is executed in the background.

The following methods are available to transfer data with the request:

• GET Request with parameters URL-encoded

• POST Request containing parameters. This is the default when an HTML <form> is used. The default content type is application/x-www-form-urlencoded. Use multipart/form-data to add any number of file uploads.

• POST Request with application type application/xml: In this case arbitrary XML is supplied (in the request body), which is passed to the parameter-transformation-input structure, inside the <input-from-request> element instead of the normal <parameter> elements.

• POST Request with application type application/json: In this case arbitrary JSON is supplied (in the request body), which is converted to XML and passed to the parameter-transformation-input structure, inside the <input-from-request> element instead of the normal <parameter> elements. Note:

  • Any characters which would be illegal in XML (for example element name starting with a digit) replaced by _xxxx_ containing their hex unicode character code.

  • Note that if any JSON objects have a key _content, then a single XML element is created, with the value of that _content key as the text body, and other keys from the JSON object being attributes on the resulting XML element.

Special Parameters

There are the following special request parameters:

Get Access to an AWS Account

Firstly you need to either create an AWS account, or you need to have access to an AWS account.

Decide on Domain

Decide on what internet domain the new Endpoints deployment should be reachable at. For example https://endpoints.myservice.com/.

This can either be a domain such as myservice.com or a subdomain such as endpoints.myservice.com. In either case, you must own the main domain you wish to host on (myservice.com in this example). Purchasing this domain is out-of-scope of this document.

Apply for an HTTPS Certificate via AWS

AWS offers free HTTPS certificates for use with their own services.

1. In the AWS Management Console (web interface), navigate to the Certificate Manager product.

2. Click “Request a Certificate”

3. On the form that is displayed, select “Request a Public Certificate” as this will be a publicly visible service.

4. Type in the domain name of the certificate; this is the full domain name, from the example above this would be endpoints.myservice.com There is no https:// part or trailing slash or dot.

5. Click on the “Review and Request” button, then the “Confirm and Request” button.

6. This will either send an email to the owner of the domain (containing a link), or it will request that a particular DNS entry is made in the DNS settings for that domain. This is to prove that you own the domain that AWS will create the HTTPS certificate for. Either click on the link in the email or set up the DNS entry. After having set up the DNS entry, a few minutes later the screen on the AWS Management Console will show the certificate has been issued.

Create VPC and Subnets

A VPC is a part of the network managed by AWS. If this AWS account will do nothing more than serve Endpoints, you can use the default VPC. If it is shared with other services, for security we recommended creating a separate VPC for the Endpoints installation. If you decide to use a default VPC, you can skip this section. If you decide to use a separate VPC, follow the following steps to create it:

1. In the AWS Management Console (web interface), navigate to the VPC product.

2. Create a new VPC, with the appropriate IP address range, such as 10.100.0.0/16. (Ignore the options about IPv6.) You can use any internal IP address range you like, as long as it doesn't conflict with any other internal IP addresses that the VPC may need to have access to (e.g. if Endpoints needs to be configured to access any databases or services internal to your company). If in doubt, talk to the department which manages IP addresses in your company. If the VPC does not need to access any other internal services, there is no reason not to proceed with the example given here.

3. Create two Subnets, with IP address schemes like 10.100.10.0/26 and 10.100.10.64/26. Allow “Specify the Availability Zones” explicitly, choose different zones for each Subnet, otherwise they all get created in the same Availability Zones. This allows services to be split across multiple Availability Zones, meaning if one AWS data center encounters issues, then the application will still be available.

4. Create an “Internet Gateway”. This will allow the VPC to send and receive requests from the internet.

   1. After the "Internet Gateway" has been created, select it, use the "Actions" drop-down to select "Attach to VPC" and select the VPC you created earlier.

5. Create a “Route Table”, then after its creation:

   1. Click on the “Routes” tab at the bottom half of the screen. Add a rule from Destination 0.0.0.0/0 to the new Internet Gateway created above.

   2. Click on “Subnet Association” tab and associate all the subnets with the routing table.

Create Security Groups

It is a good idea to create all security group in advance. Security groups can also be created when databases and other resources are created, however then they have useless names such as “rds-launch-wizard-2”.

On the AWS Management Console (web interface), navigate to the Security Group feature. The following table specifies which security groups to create. Each have a name and one or more inbound rules. For outbound rules, allow the default which is to be able to send requests on any port to any location.

Create "Bastion" VM

For various tasks, it is useful to have a VM to connect to via SSH. This is "behind the firewall" and will allow you to access resources such as the database which are not public.

1. In the AWS Management Console (web interface), navigate to EC2 product.

2. If you do not already have an SSH public key then go to the left navigation under "Key Pairs" and create a new Key Pair. If you already have an SSH public key, e.g. created on your computer, you do not need to do this step, you can upload it later.

3. Navigate to the "Instances" section of the left-hand navigation.

4. Click “Create New VM”

5. Select the latest Ubuntu image

6. Select a very small instance size (to save costs)

7. Select the VPC created earlier

8. Set “Auto-assign public IP” to Enable

9. Set the tag “Name” to e.g. “SSH from Internet”

10. Select the “SSH from Internet” Security Group created earlier

11. Select an SSH Key pair that you have access to so that you can log on to the server.

It takes quite a while before a user can log in to a newly created EC2 instance, for example 5 minutes. You might see the error "Permission denied (publickey)" during this time.

To connect to it, use the ssh username ubuntu together with the key you created or uploaded earlier.

Create PostgreSQL Database

Endpoints uses the database to store various things such as which Endpoints "applications" have been installed.

1. In the AWS Management Console (web interface), navigate to RDS product. This is the AWS managed database product.

2. Click “Create a new database”.

3. Name it something like “Endpoints”.

4. We currently support PostgreSQL 14 (although other versions will probably work).

5. Select a random master password.

6. Select “Create new Subnet Group”

7. Set “Public Access” to “No”, as this database should not be publicly visible on the internet.

8. Select the database Security Group that was created earlier.

9. Database schema creation is not necessary; software does that on startup.

10. “Point-in-Time Recovery” is activated by default, so no action is required to enable that.

11. Make sure that “Auto Minor Version Upgrade” is enabled, so that you do not have to take care of manually upgrading the database between minor versions.

Create the Task Definition

A Task Definition is a blueprint of how AWS will install and run the Endpoints software.

1. In the AWS Management Console (web interface), navigate to the ECS product. ECS is the service AWS offers to manage Docker installations.

2. Create a new Task Definition

3. Select type "Fargate". This means that AWS itself will automatically allocate compute resources, as opposed to having to do it manually.

4. Name the Task Definition with a name like "Endpoints"

5. Select the RAM and CPU. We recommend at least 500MB RAM.

6. Add one container within the Task Definition. Give it a name like "endpoints".

7. The URL to the public Docker image of Endpoints is public.ecr.aws/x1t6d0t7/endpoints-he

8. Set a hard memory limit e.g. 450MB RAM.

9. Add a single port mapping to the container. Container port is 8080.

10. 1. ENDPOINTS_JDBC_URL is like jdbc:postgresql://xxxxx/postgres?user=postgres&password=postgres where xxxx is the host value of the RDS database

    2. ENDPOINTS_BASE_URL is the URL of the service e.g. https://endpoints.myservice.com/ with a trailing slash

    3. ENDPOINTS_SERVICE_PORTAL_ENVIRONMENT_DISPLAY_NAME For example “live environment”. This is just text which is displayed on the Service Portal login page. In case you have multiple Endpoints installations, it is convenient to differentiate them with text on the login page.

Create an ECS Cluster

A "Cluster" is a set of compute resources, managed by AWS, where Endpoints will run.

1. In the AWS Management Console (web interface), navigate to the ECS product

2. Create the ECS cluster (not Kubernetes Cluster)

3. Select “Networking only” type.

4. Don’t select creation of a new VPC

Create a Load Balancer (ELB)

The Load Balancer is responsible for taking the user's HTTPS requests, and forwarding them on to the Endpoints software running on a managed ECS cluster created above.

1. In the AWS Management Console (web interface), navigate to the ECS product

2. Go to the “Load Balancers” section.

3. Click “Create Load Balancer”.

4. Select the default “Application Load Balancer” from the two options.

5. Change the listener to be HTTPS only.

6. Select the correct VPC, which was created above.

7. Select all subnets.

8. Select the HTTPS certificate that has been previously created.

9. Select the HTTPS security group previously created.

10. Go to the "Load balancer target group".

    1. Create a new Target Group. However, its settings are not important, later it will be deleted, as each time a Docker instance is registered with it, it will create a new Target Group.

    2. Do not register any targets to the newly created Target Group (as it will be deleted later)

Connect the Domain to the Load Balancer

This is necessary so that when someone navigates to your domain, their requests are sent to the AWS Load Balancer created above, and thus the request can be served by Endpoints.

1. In the AWS Management Console (web interface), go to the EC2 product.

2. In the Load Balancer section, click on the Load Balancer created above. You will see it has a DNS name such as Endpoints-HTTPS-66055465.eu-central-1.elb.amazonaws.com (A Record).

3. In the tool where you administer the domain, create a CNAME DNS record, from the domain or subdomain chosen for this installation, to the domain name you can read off in the last step.

Add Services to the Cluster

This step takes the Task Definition you have created earlier (which is a blueprint for running the software) and installs it on the Cluster created earlier (a set of compute resources).

1. In the AWS Management Console (web interface), go to the ECS product.

2. Navigate to ECS “Clusters” (not Kubernetes Clusters)

3. Select the newly created “Cluster”.

4. Create an service called “Endpoints”

5. Type is Fargate

6. select the Task Definition created above

7. Select the VPC created above

8. Add all subnets

9. Select the “webapp” security group, created above

10. In the Load Balancing section:

    1. Select “Application Load Balancer”

    2. Select the load balancer previously created in the drop-down

    3. Click the “Add to load balancer” button

    4. Select the target group name

    5. Select the existing “production listener”

    6. The URL is /* i.e. slash followed by a star

    7. Health check is /health-check

11. Set the application as "sticky" in the load balancer: (This is required for the Service Portal inc case more than one instance is running as the "state" of the web application is stored in the server memory)

    1. Navigate to the EC2 Product in the Management Interface.

    2. Go to the Target Group section in the left navigation

    3. Select the previously-created Target Group

    4. Navigate to the "Attributes" tab

    5. Click "Edit"

    6. Enable the "Stickiness" checkbox

    7. Select the "Load balancer generated cookie" option.

Create Monitoring Alarms

It is possible, but not necessary, to use CloudWatch to create monitoring alerts for the health of various components such as the database. To create an alarm:

1. In the AWS Management Console (web interface), go to the CloudWatch product.

2. Click “Create Alarm”.

3. Set up the alarm, as described below

4. On the last screen, select that the action should be to send an email to the appropriate person.

Perform the above steps for each of the following alarms:

• CPU: ECS -> CPUUtilization, > 70 (percentage) for 1 period (= 5 minutes)

• Memory: ECS -> MemoryUtilization > 70 (percentage) for 1 period (= 5 minutes)

• Up: ApplicationELB > Per AppELB, per TG Metrics -> UnHealthyHostCount, > 0 for 1 period (= 5 minutes)

• DB Disk: RDB -> Per-Database Metrics -> FreeStorageSpace for digitalservices < 5 (percentage) for 1 datapoint

Use Application

The application is now available under your URL e.g. https://endpoints.myservice.com/

If the application is configured in multi-application mode then the Service Portal is available under https://endpoints.myservice.com/service-portal with username admin/admin.

Troubleshooting

• Go to the ECS Cluster, go to the Service, click on the “Events” tab to see what’s going on, e.g. health check is failing

• Go to CloudWatch, Log Groups, see that there is a new log group which has been created, go into the log file and see if there are any errors.

The term ”application” is used to describe a piece of configuration. OpenEndpoints can serve multiple applications simultaneously. You can create different projects with different application directories and run them in parallel.

An application is configured with a directory of files - mainly XML files. The structure of the directory is explained in detail here: .

OpenEndpoints pulls the application directory from a Git. When publishing, the last version is pulled from the Git and your configuration is updated.

Add or Edit Application

Click Add New Application to create your first application.

This will open the screen to add new or edit existing applications:

• Choose an application name, which will become part of the endpoints URL

After successfully adding a new application you will find your new application in the "Change Application" Screen:

OpenEndpoints installed in "Multi Application Mode" (see ) comes with a service portal for administration:

https://[your-base-url]/service-portal

Note that in "Single Application Mode" your application is already packed with the Docker container.

Default Login

A default account for the admin is automatically created during installation:

Open the application you want to publish:

Publish to Preview

This option will:

1. Load the latest available revision from your Git repository

2. Check the new configuration for consistency and errors. If an error is found, the new version will not overwrite any existing status. A detailed description of the error is displayed.

3. On success the new revision will replace the previous configuration.

Promote Preview to Live

This option will not pull a new version from the Git, but simply copy the existing configuration from Preview to Live. No further check of consistency is required, as this had been done already when publishing that version to Preview.

Description

This software takes XML from multiple sources (e.g. URLs, files, or any Java class you write which can produce XML). You configure what sources you want, and the software combines the XML from those sources into one XML file in memory. The XML is then optionally transformed using XSLT. The resulting document is then optionally converted into PDF, XLS or JSON. The resulting file is then either downloaded to the browser, or a sent as an email, or you can write a Java class which performs any action on the file, or any combination of the above.

Documentation on Gitbook

A documentation of our software is available here: https://openendpoints.gitbook.io/

We would like to thank GITBOOK for providing us with their great product for free.

Further documentation is available in the LyX doc . Download LyX for free for Windows/Mac/Linux in order to read and contribute to this file.

Authors

This software was originally written by and commissioned by . At the time of writing, Offer-Ready IT-Services & Consulting GmbH still maintain the project.

Contributing

Contributions are welcome. Please open an issue describing what you wish to achieve. We will be able to help you with advice, before you invest the time of development. When you've developed your patch, please submit a pull request using github.

This is an example installation on . For sure there are many more paths how-to install OpenEndpoints on DigitalOcean.

1. Sign up to DigitalOcean or be invited to an existing account

2. On the left hand navigation go to “API” and “Generate New Token” with a name such as your name. Record the token somewhere for later.

3. Create a Project

   1. Top-left of the navigation all projects are listed

   2. Click on “New Project”

   3. Enter the name “Endpoints”

   4. For “What’s it for” answer “Operational / Developer tooling”

   5. For “Environment” answer “Production”

4. Upload the Endpoints software as a Docker image to DigitalOcean Docker Repository

   1. Go to “Container Registry” on the left hand side navigation

   2. Use a name which is unique to your company or product; this is globally unique across all of DigitalOcean. We use “endpoints” in this example.

   3. On your local computer (e.g. Mac): Install "doctl", see https://docs.digitalocean.com/reference/doctl/how-to/install/

   4. doctl auth init

   5. doctl registry login

   6. docker pull public.ecr.aws/x1t6d0t7/endpoints-he

   7. `docker tag public.ecr.aws/x1t6d0t7/endpoints-he registry.digitalocean.com/endpoints/endpoints

   8. docker push registry.digitalocean.com/endpoints/endpoints

5. Create a database:

   1. Click on “Databases” in left navigation

   2. Click "create"

   3. Choose “PostgreSQL”

   4. Currently we recommend PostgreSQL 14 (although other versions will probably work).

   5. We recommend starting with the cheapest version which is “Basic Node” with 1GB of RAM.

   6. Choose your Data Center.

   7. Under “Choose a unique database cluster name” choose something like “endpoints”.

   8. Click the green “Create a Database Cluster” at the bottom of the screen to actually start the creation process. (The creation process takes a while, e.g. 5-10 minutes.)

   9. After you start the creation process, the resulting screen displays information about the database, with a few tabs such as “Overview”, “Insights” etc.

6. Create the application:

   1. On the left navigation click on “App Platform”

   2. Create a new app

   3. Select "source provider" of “DigitalOcean Container Registry”

   4. Add the following environment variables:

      1. JAVA_OPTIONS is -verbose:gc

      2. ENDPOINTS_BASE_URL- Use whatever domain you want the service to be running under e.g. https://endpoints.mycompany.com/

      3. ENDPOINTS_JDBC_URL

         1. Go to the database settings screen

         2. Go to the bottom right of the screen “Connection Details”

         3. Select the default “Public Network”

         4. Use a format like jdbc:postgresql://<host>:<port>/<database>?user=<username>&password=<password>

      4. ENDPOINTS_SERVICE_PORTAL_ENVIRONMENT_DISPLAY_NAME is e.g. DigitalOcean

   5. Choose an app name, this is used in the internal URL but otherwise doesn’t matter much

   6. Wait for it to deploy

   7. See the URL and check it works

   8. Enter a CNAME in your DNS from the URL you want to the one that DigitalOcean has supplied

   9. Go to Settings, in “App Settings” under “Domains” add the domain you want the service to run under, so that HTTPS certificates work.

A simple web form to send data to OpenEndpoints could look like this:

What shall happen after submit?

The form calls your endpoint, which might for example

The behaviour of your web form after pressing the submit button depends on both the actions defined in your endpoint (success and error actions), and the html code in your web form.

Option 1: Redirect to Success-Page / Error-Page

On success (but not on error) you may use parameter placeholders. For example, you could add tracking information that is executed by your success page.

Option 2: Return HTTP-Status

Omitting a particular action on success or failure will simply return a status code of 200 for success or 400 for any type of failure.

Option 3: Return Custom JSON

To output JSON, the XSLT should use the output method "text":

The transformer should explicitly define the content type "application/json":

Option 4: Download Document

Form Inputs vs Endpoint Parameters

Option 1: Endpoint With Parameter Transformation

• The name attributes in the web form can be completely different from the names of the endpoint parameters.

• The parameter-transformation-xslt may only output parameter values of parameters that are declared in the endpoints.xml file. There are no restrictions as to how inputs from the web form are mapped with this output.

• The output generated by your XSLT requires values for any parameter not having a default value. Not doing so will raise an error.

Option 2: Endpoint Without Parameter Transformation

• Input from the web form is passed to endpoint parameters only if the name attribute in the web form matches the name attribute of the endpoint parameter. Note that names are case sensitive.

• Parameters without a default value must have an equivalent in the web form. Only parameters for which there is a default value are optional.

• Inputs from web forms for which there is no corresponding parameter are silently ignored.

After you've published your application, you probably want to test your endpoint :-)

Calculate Hash

1. Navigate to Calculate Hash in the main navigation.

2. Select your endpoint and the environment.

3. In case your endpoint has "include-in-hash" parameters, you will be prompted to enter values for those parameters. See: .

4. Press button Calculate Hash and copy the calculated hash value.

Build the Link

1. Navigate to Home in the main navigation.

2. Copy Live URL or Preview URL - depending on what environment you would like to use. The URL should look like this: https://endpoints.openendpoints.com/foo/{endpoint}?{parameter}

3. Replace {endpoint} with the name of your endpoint

4. Replace {parameter} with: hash=[the-calculated-hash]

5. If your endpoint has "include-in-hash" parameters, those parameters (and the values as used for calculating the hash) shall be added as additional GET parameters.

Test the Link

1. Copy the link into your browser.

2. Navigate to Request-Log in the main navigation. You will find your request, including additional information in case an error has occurred.

Even on a static website there is always content that changes regularly and can be loaded from a database or a web service, for example. OpenEndpoints is perfect for converting external content into HTML, which can be seamlessly integrated into your own website.

HTML Sniplets

Build an Endpoint to Return XSLT Transformation. Do not use the download-filename attribute in this case.

The generated HTML is inserted directly into the <div>. Therefore, the HTML (generated with XSLT) should not contain a <head> or <body> tag, but actually only the content that is to be inserted!

Set content-type "text/html" in the transformer:

<transformer data-source="...">
    <xslt-file name="..."/>
    <content-type type="text/html; charset=utf-8"/>
</transformer>

Load Snippet into Web Page

Java Script can be used to load the html returned from the endpoint into a <div>.

In this example the script was taken from https://www.w3schools.com/lib/w3.js.

<html>
  <head>
    ...
    <script>
      function includeHTML() {
        var z, i, elmnt, file, xhttp;
        /* Loop through a collection of all HTML elements: */
        z = document.getElementsByTagName("*");

        for (i = 0; i < z.length; i++) {
          elmnt = z[i];
          /*search for elements with a certain atrribute:*/
          file = elmnt.getAttribute("w3-include-html");

          if (file) {
            /* Make an HTTP request using the attribute value as the file name: */
            xhttp = new XMLHttpRequest();
            xhttp.onreadystatechange = function () {
              if (this.readyState == 4) {
                if (this.status == 200) {
                  elmnt.innerHTML = this.responseText;
                }
                if (this.status == 404) {
                  elmnt.innerHTML = "Page not found.";
                }

                /* Remove the attribute, and call this function once more: */
                elmnt.removeAttribute("w3-include-html");
                includeHTML();
              }
            };
            xhttp.open("GET", file, true);
            xhttp.send();

            /* Exit the function: */
            return;
          }
        }
      }
    </script>
  </head>
  <body>
    ...
    <div w3-include-html="[path-to-your-endpoint]"></div>
    ...
    <script>
      includeHTML();
    </script>
  </body>
</html>

Each request to OpenEndpoints requires a mandatory hash parameter:

The hash is a SHA-256 hex calculated from several input values. On each request the server calculates the expected value of the user-supplied hash parameter. If the supplied value does not match the expected value the request will be denied.

The hash may be supplied as uppercase or lowercase in the request.

Secret Key(s)

In the security.xml file under application you have to create one or many secret keys.

<security>
    <!--it is mandatory to have at least 1 secret key-->
    <secret-key>any-string</secret-key>
    <!--optionally you can define any mumber of additional secret keys-->
    <secret-key>optional-another-string</secret-key>
</security>

Any secret key in security.xml will be a valid input to determine the expected calculated hash. The advantage of having more than one secret key is that you can implement a rotation of secret-keys without interruption of services:

1. Add a new secret key

2. Adopt new hash values in your applications or web forms (using the new secret key)

3. Delete the old secret key

Calculation of the Hash

Input of the SHA256 function is the concatenated string of the following inputs:

1. Name of the endpoint

2. The values of all the parameters listed in the <include-in-hash> block of the endpoint, in the order in which they are listed there. (If there is no <include-in-hash> section, or there is but it's empty, then no parameters are added to the hash's source string for this step.)

3. Environment name (either “live” or “preview”)

4. Any secret key from the security.xml file

Example Calculation

<security>
    <secret-key>openendpoints</secret-key>
</security>

<endpoint name="helloworld">
    ...
    <include-in-hash>
       <parameter name="foo"/>
        <parameter name="long"/>
    </include-in-hash>
      ...
</endpoint>

Assume that

# parameter name="foo" value="abc"

Include-In-Hash Use Cases

The possibility to include parameter values in the hash calculation can be used to implement use cases like:

• Send a link to web form having some distinct mandatory input which shall not be changed. Changing the value of that parameter would result in a different expected hash value. Hence, the form will only work with the (unchanged) value.

• Having the same endpoint implemented for different web forms or different applications could use a hidden parameter indicating the originating source. The expected hash can be different for each originator, making it impossible to modify that value.

• Generate the hash in combination with a timestamp. You can invalidate the request after a certain time - and the timestamp can not be modified because this would change the expected hash value.

HTML web forms allow to send files along with the form data:

<input type="file" name="foo" />

<!-- It is possible to upload multiple files using the multiple attribute: -->
<input type="file" name="foo" multiple="true" />

OpenEndpoints automatically understands the correct enctype to separate uploaded content from form <input> data:

Once the file(s) have been uploaded by the user, the following options exist to do things with these files:

Option 1 - Email attachment

The Email Task allows to add all uploaded files as attachments:

<attachments-from-request-file-uploads/>

That means: It is not possible to select which file should be attached as an attachment. All files are always attached.

Option 2 - Insert file content into HTTP request body

When sending an HTTP request from OpenEndpoints using the HttpRequest Task , and when specifying that HTTP request should contain an XML body, and when that XML is specified inline, the optional attribute upload-files="true" may be set.

For example:

<xml-body upload-files="true">
    <foo upload-field-name="xyz" encoding="base64"/>
</xml-body>

This syntax indicates that the element <foo> will be filled with the contents of the uploaded file with the filename "xyz". The encoding is always base64, no other encodings are supported.

A distinction must be made between two technically different topics:

• The web form containing multiple controls having the same name attribute.

• The web form containing a single control, but which allows multiple values to be selected.

Multiple controls having the same name attribute

According to w3c standard HTML forms can have multiple controls sharing the same name attribute. See: https://www.w3.org/TR/html52/sec-forms.html

For example:

<form>
  <input type="checkbox" name="topping" value="bacon" />
  <input type="checkbox" name="topping" value="cheese" />
  <input type="checkbox" name="topping" value="onion" />
</form>

OpenEndpoints will combine all values submitted into a single input-from-request parameter, separated by a multiple-value-separator. The default separator is a double pipe ||.

The above example would result in a parameter "topping" that has the following value:

<parameter name="topping" value="bacon||cheese||onion"/>

It is possible to define any other multiple-value-separator. See: Multiple parameters supplied with the same name.

HTML input types allowing multiple values

Some HTML form controls allow to submit multiple values for a single control, for example:

<select name="topping" multiple="true">
  <option value="bacon">Bacon</option>
  <option value="cheese">Cheese</option>
  <option value="onion">Onion</option>
</select>

On selecting multiple values, this would be submitted by the user agent in the same way as having multiple controls with the same name attribute. The result will be the same as above in OpenEndpoints:

<parameter name="topping" value="bacon||cheese||onion"/>

Uploading multiple files

It is possible to upload several files at the same time:

<input type="file" name="foo" multiple="true" />

See: Web Form With File Upload

The optional xml-from-application directory contains custom xml files that may be used as a data-source. There are no restrictions on how these files should be structured.

The idea behind locally stored content is to avoid unnecessary data load if content doesn't change often.

The optionalstatic directory can be used to store any non-XML local content, for example images and prefabricated PDF. You may use subdirectories.

Images in PDF

On inserting images into PDF, the static directory serves as the local root path to the XSLT processor. So you can either use an absolute path (URL) or a relative (local) path that refers to this directory.

Images in Email Body

If the email-body has a content type like text/html; charset=utf-8 then it may include tags such as <img src="cid:foo/bar.jpg">. The tag is most commonly an <img>, but can be any tag. The system then searches in the static directory for any file with that path. The file is included with the image, as a “related” multi-part part, meaning the file is available to the HTML document as an "inline image" when its rendered in the email client. Don't forget the cid: text in the src attribute!

Return Static Files

Instead of returning content from XSLT transformation, you may also return “prefabricated” content from this directory. The syntax <response-from-static filename="path-to-file"/> returns a file from this directory.

In general <error> supports the same tags as <success>. But there are limitations.

No forward-to-endpoint

Placing <forward-to-endpoint> inside <error> is not supported, as this would require “auto increment” numbers to be assigned even in the case of <error> (as any arbitrary endpoint might be called, which might require them). It is an explicit design goal to only support auto-increment numbers (e.g. for invoice numbers) in the case of <success>.

Limited Support of Variables

On error the same tags can be used as for <success>, but placeholders for client generated parameter values are not supported. An error might have happened during parameter transformation, and therefore client generated parameters are not necessarily available.

System generated parameters may be used.

In addition, for the processing of the <error> tag the following additional parameters are available:

• ${internal-error-text} - This contains an internal error message. It is important this is not exposed to any end customer, as it might contain security-sensitive information such as “cannot connect to database at IP address 1.2.3.4” etc.

• ${parameter-transformation-error-text} - In case the request failed because the parameter transformation failed, and a message was set in the <error> tag in its output.

By default, OpenEndpoints does not save any data that was transferred with a request.

Sometimes, however, it can make sense to have data transfer tracking available for error analysis, for example. This is what the debug mode and verbose is for.

Working Principle

With the parameter transformation, OpenEndpoints generates a parameter-transformation-input.xml and - after successful transformation - a parameter-transformation-output.xml. These are only stored in memory and not persisted to disk, by default.

When Debug Mode is activated, these two files are saved in the request log and can be downloaded from the Service Portal. (If the transformation fails, the output.xml is not generated).

Usage

To use the debug mode, either 2 conditions must be met at the same time:

1. The debug mode must be allowed for this application in the service portal.

2. An additional parameter is added to the request: debug=true

or

1. The verbose=true parameter must be sent

2. The request has an error (4xx or 5xx)

The intention of verbose=true is to capture errors, therefore verbose does not require the application have "debug enabled", in contrast to sending debug=true which captures all requests and thus does require the application to have "debug enabled".

Clear Debug Log

Click "Clear debug log" to delete all files created with debug/verbose mode.

The current configuration of an application is loaded from a GIT to OpenEndpoints (with the "Publisher" in the service portal).

To enable a simple form of "staging", 2 different versions of the GIT repository can be activated at the same time. There are 2 different "environments" for this: PREVIEW and LIVE.

Publisher

The "Publisher" in the service portal lets you

1. Publish the latest version of the GIT repository to the PREVIEW environment.

2. Promote the current version from PREVIEW to LIVE.

Usage

By default, all requests are always sent to the LIVE environment. To send a request to the PREVIEW environment, an additional parameter must be added:

It is not required to add a parameter for the LIVE environment.

The optional file email-sending-configuration.xml under the application directory is required if emails are to be sent from OpenEndpoints.

OpenEndpoints supports two options for sending emails:

• Sending via SMTP

• Sending via MX address

The optional directory data-source-post-processing contains custom XSLT files.

See: .

Transformation Input

The root tag of the input xml (which is automatically provided by the software) is <data-source-post-processing-input>. Within that root tag OpenEndpoints will insert the data converted from the data-source-definition or - if applicable - the output from a previous data-source-post-processing step.

Transformation Output

The root tag of the output (=the file that is generated by your XSLT) must be <data-source-post-processing-output>.

The optional file service-portal-endpoint-menu-items.xml enables the creation of additional, user-defined pages in the service portal.

Pages of type form and content can be optionally organized under a menu-folder. The maximum supported depth of menu items is one.

Form

The form consists of two pages: the "form" to enter data, and the "result" of the form submit. Both pages are defined by a transformer returning an HTML page with mime-type application/html.

The "form" (=the page returned by calling the form-endpoint) may contain form elements, but without a <form> tag and without a submit button. The form data are submitted to the result-endpoint.

Content

The content consists of a single page defined by a transformer returning an HTML page with mime-type application/html.

The entire configuration of an application that is built with OpenEndpoints consists of files in a directory - we refer to this as the Application Directory.

An example directory structure is available on Github (public repository):

For each configuration file or directory, additional explanations are available in the subsections of this page.

Note that is not a working example application. The aim of the example directory is to present the expected syntax and directory structure.

Application loaded from Git

The application directory resides in your own Git repository. There are 2 :

• Multi Application Mode (=default): The Service Portal provides a simple user interface to "publish" the latest version from Git into the software. An integrity check with meaningful error messages is carried out automatically. Most configuration errors are detected in this way.

• Single Application Mode: You create a new Docker image derived from the standard Docker image, which includes the application directory. You need to build a new Docker image each time your configuration has changed.

The multi-application mode also offers the possibility of simple staging - see .

Expected XML sometimes less strict than XSD

The software is a little less strict than what is written in the XSD: Don’t be surprised if you find working examples where the order of elements is slightly different from what is described in the XSD.

Additional files silently ignored

Any additional files which are present in the application directory, but not required by the software, are silently ignored. For example, if you create an additional directory "project files" that will be ignored by the software without raising an error.

Minimum configuration requirement

Many configuration files are only required if the corresponding features are used.

In any case, these few files must be present for a working configuration:

• endpoints.xml having at least 1 endpoint

• security.xml having at least 1 secret key

• at least one transformer, which as a matter of facts also requires a data-source

The mandatory file security.xml under the application directory comprises of one or many secret keys, which are used to calculate hash keys. See:

The mandatory file endpoints.xml under the application directory comprises of one or many endpoint folders, each of which contains of

• one or many endpoint definitions

• zero or many

Endpoint Folders

The root tag of the file is <endpoint-folder>. A simple endpoints.xml might have a single <endpoint-folder> only (in this case it is the root tag itself). But it is also possible to create a hierarchy of endpoint-folders:

Parameters may be defined at any level. If they are defined in an Endpoint they only apply to that Endpoint. If they are are defined in a folder they apply to all Endpoints under that folder.

Settings in the child folder will override the parent settings.

Parameters

Parameters require to have a name attribute, which is unique within the endpoint-folder. The default value is optional, but using will change the behaviour of the application. See Endpoint Parameter.

Note that a GET or POST request allows multiple parameters with the same name. For example, a web form might have 2 or more form fields with the same name attribute. In contrast to this, the name attributes of the endpoint parameter must be unique. As a consequence - if parameter names are used multiple times in GET or POST requests - several values would be transferred for the same endpoint parameter. The solution is to separate multiple parameter values with a separator. The default separator is a double pipe ||. The attribute multiple-value-separator allows to define an alternative separator.

Endpoint

The basic syntax of any endpoint is:

Details for each property can be found in the related sections of this documentation:

• Parameter Transformation allows to optionally transform submitted parameter values, for example to validate, normalize or correct submitted values or to create new parameters by calculation or by loading additional data from any data source.

• Include-In-Hash Use Cases allows for advanced authentication requirements based on parameter values.

• Section Types of Endpoints describes all action types for <success> and <error>.

• Tasks include actions such as sending an email, sending a request to any kind of REST-API or logging data into a database.

The optional directory data-source-xslt contains custom xslt files.

Each transformer references to a data-source, which can be transformed to any output having some custom data-source-xslt file.

See:

Transformation Input

The data-source will generate a file with a root tag <transformation-input> comprising of all content generated by the data-sources.

The mandatory directory data-sources contains data-source definitions. A data-source-definition contains references to one or many data-sources, which you would like to load data from and convert to output using a transformer.

See: .

OpenEndpoints can interact with third party REST or SOAP APIs, either using such API as a data-source, or to execute a TASK.

The request body of such request can be either inline content (=directly written into endpoints.xml or into the data-source definition), or the body is generated by XSLT transformation.

This optional directory contains custom xslt to generate such request bodies.

The transformation input (=the XML generated by the software, which the XSLT is applied to) does not support arbitrary data-sources, but is limited to parameter values:

The values of the parameters inside that generated xml are different depending on the context:

Request Body Within Parameter Transformation

The input xml contains the parameters from the original request. Note that the original request could have different parameters than those declared in the endpoints.xml - which is possible when parameter transformation is used.

Request Body Within Data-Source or Task

HTTP requests that are part of a regular data-source definition or a task have the same parameters and parameter values in the input xml for generating a request body that are declared in endpoints.xml and whose values can be different after a parameter transformation than in the original request.

You can embed custom fonts into your generated PDF.

In order for custom fonts to be embedded in a PDF, 2 steps must be set:

1. Upload the TTF files into the fonts directory. Only TTF fonts are supported.

2. Declare those fonts in the file apache-fop-config.xsd.

apache-fop-config.xml

Every combination of style and weight of a font must be declared as a font triplet in file apache-fop-config.xml:

Note that the file-name of the TTF font may be different from the name as used in the font-triplet attribute. The font-name as referred to in the XSLT must be the same as in the font-triplet name attribute (case sensitive).

The mandatorytransformers directory contains one or many transformers defining a .

The only mandatory attribute is the data-source. Omitting all other tags will output the original data-source (xml) without any transformation.

This optional directory contains custom XSLT for .

Parameter Transformation Input

If an endpoint uses the parameter transformation, OpenEndpoints automatically generates the temporary XML parameter-transformation-input.xml, which is the input for the transformation. This is (unless debug is enabled) only stored in memory, and never written to disk. Depending on how data are submitted to the endpoint, the input XML might look slightly different.

• In case input was submitted as GET request or POST request containing parameters, the generated parameter-transformation-input.xml will include submitted parameters.

• In case data are submitted as a POST Request with application type application/xml, the input-from-request tag contains the XML request body instead of parameters.

• In case data are submitted as a POST Request with application type application/json, the input-from-request tag contains the json request body converted to xml:

Parameter Transformation Output

The expected output (generated by your custom xslt) requires values for any parameter not having a default value. Not doing so will raise an error. In addition the output might have an error tag. If present, an error will be raised an the text within the error tag will be available with the system parameter ${parameter-transformation-error-text}.

A request contains any number of GET or POST parameters.

The values supplied by the user for these parameters are used in various places. For example, if you are fetching XML from a URL, parts of the whole of the URL may be the value of a parameter. The syntax for this is ${param}. The braces are, in contrast to many other systems, mandatory.

Note that for security or technical reasons replacement is not always possible. See the specific sections for the various configuration elements where parameter replacement occurs.

Parameters must be defined in the endpoints.xml file

Which parameters the user may supply must be defined in the endpoints.xml file; any parameter sent to the server which is not defined there is an error.

The simplest form of a parameter definition in endpoints.xml is <parameter name="foo">. This means the user must supply a value that parameter as a GET or POST parameter like ?foo=value in the request; not doing so is an error. (A request parameter transformation allows to submit parameters not defined in the request. For details please refer to section .)

Adding a default value makes the parameter optional. If the user doesn't supply anything, the default value will be used.

Parameters are defined, in the simplest case, directly under the root <endpoint-folder> node of endpoints.xml. Parameters under the root <endpoint-folder> node are available for all endpoints defined in the file.

Multiple parameters supplied with the same name

The HTTP standard allows multiple GET parameters to be supplied with the same name, like

In this case, all those values are concatenated into the same parameter. The default separator is two pipe characters ||.

In the example above, the parameter value would be foo||bar.

The default separator be overridden with the multiple-value-separator attribute:

Note that this property will be inherited in the hierarchy of endpoint-folders. That means it can be overwritten by subordinate folders or endpoints.

System Parameters

There are certain parameters which are always available, and are not provided by the client. These can also be used with the same ${param} syntax.

• ${request-id} - this system parameter returns the globally unique id of the request, assigned by OpenEndpoints during the request.

AWS Secrets

It is possible to load secrets from the AWS Secrets Manager and make them available via ${foo} parameters, for example in the SMTP configuration.

The relationship between ${foo} parameters, and which secret they are backed by, is specified in the optional secrets.xml file, checked in at the root of the application directory.

Note, not all secrets need to actually need to exist in the AWS Secrets Manager. An error is produced only if the parameter (and thus the secret) is actually used. This is to allow “if” statements in the configuration, executing certain commands only on certain environments. If the configuration is executed on a different environment, the secret need not exist, as that part of the “if” statement is never executed.

Secrets are fetched each request. This allows for easy secret rotation. No caching of secrets between requests occurs, and no restart of the application is necessary in the case of updating a secret within AWS Secrets Manager to a new value.

JSON-style secrets, which the AWS Management Console web application creates by default, are not supported. The value of a secret is assumed to be plain text.

This type of syntax specifies that a redirect of the request should be performed. The body of the <redirect-to> specifies where. Variables, if present, are replaced, in the body.

For example you can redirect a successful request to a "thank.you.html" url.

Parameters like ${foo} in the body are replaced.

If you use variables, we recommend to use this optional element to prevent a malicious request redirecting somewhere wrong:

If no such tag is present, redirect to any URL is allowed. If one or more are present, the URL being redirected to must start with the prefix of one of them; otherwise this is an error.

This type of syntax specifies that the response content is fetched from the named file within the static directory. Subdirectories such as filename="sub-dir/file.pdf" are supported.

Variables are not allowed in the filename attribute. This is to prevent accidentally making more files accessible (if the client guesses the right filename) than intended. But you can use to deliver different files depending on different parameter values.

You can optionally add an attribute "download-filename".

If the download-filename="foo.pdf" attribute is present, then the header in the HTTP response is set indicating that the file should be downloaded as opposed to displayed in the browser window. Parameter Values like ${foo} are replaced.

This type of syntax specifies that an HTTP request is performed and the result of this request is streamed back to the client as the response of the call to Endpoints.

Syntax

You can optionally add an attribute "download-filename".

If the download-filename="foo" attribute is present, then the header in the HTTP response is set indicating that the file should be downloaded as opposed to displayed in the browser window. Parameter Values like ${foo} are replaced.

In endpoints.xml, <endpoint> has two sub-sections, <success> and <error>. What happens in the case of success or error depends on the tags being present. Details are described in the following subsections.

If the tag (<success> or <error>) is missing, or present and empty, this means the server returns an empty 200 OK in the success case and 400 error in the case of failure. This can be useful if the request should simply perform some tasks e.g. send emails.

This type of syntax specifies that the transformation named in the attribute is performed, and its contents are returned to the client in the HTTP response. For example, the transformation might produce HTML to be displayed in the browser, or a PDF to be downloaded.

Syntax

You can optionally add an attribute "download-filename".

If the download-filename="foo.pdf" attribute is present, then the header in the HTTP response is set indicating that the file should be downloaded as opposed to displayed in the browser window. Parameter values like ${foo} are replaced.

Basic Principle

<OpenEndpoints/> transforms content from CMS, REST-APIs, databases and static files into other content structure and other content types. The technical solution is XML Transformation.

This requires

1. XML as a data-source

2. XSLT to transform XML into other content structure and content type

XSLT (Extensible Stylesheet Language Transformations) is a language for transforming XML documents into many different formats: other XML documents, or other formats such as HTML, PDF (precisely: XSL-FO), or plain text which may subsequently be converted to other formats, such as JSON. XSLT is an open, stable and established technology. Read more here: .

While XSLT is incredibly versatile and powerful, XML is not a really common type of data-source in the web - which we sincerely regret, because XML is just perfect to describe all sort of semantic content and provide established tools to manipulate that content. This is where <openEndpoints/> comes in:

Required components are:

1. The data-source

2. The XSLT

3. The "transformer" - which basically combines the data-source and the XSLT in order to generate new output.

Data-Source

In the data-sources directory under the application, there are zero or more files, each describing a data-source.

A data source is a list of commands (e.g. fetch XML from URL) which produce XML. Each data source is executed, and the results are appended into an XML document (e.g. fetch XML from two URLs, then the result of the data source will be an XML document with two child elements, which are the XML fetched from the two URLs).

The data source file contains the <data-source> root element then any number of data-source command in any order:

The resulting XML document has the root tag <transformation-input>. The results of the command are appended, in order, directly underneath this tag.

Data-Source XSLT

The XML Transformation is stored in a single file ("xslt-file"). In the data-source-xslt directory under the application, there are zero or more XSLT files than can be used for your transformations. You can use subdirectories to organize your xslt files.

The data-type of the generated output is determined by the XSLT file.

Transformer

In the transformers directory under the application, there are zero or more files, each describing a transformation. The transformation determines which XSLT to apply on which data-source.

In contrast to content loaded from the internet, XML already existing within the application does not require to be loaded each time. This makes it a good choice for large files. It is possible to use XML files with more than 100,000 lines of content without causing any performance issues. In general, the limiting factor is the performance of the XSLT transformation rather than the size of the data source.

You can place such files under the the xml-from-application directory within the application. You can load the content of this file into the data-source:

You can use placeholders to fill in endpoint parameter values into the file attribute:

If the file can not be found, an error will be raised. To avoid that, add an attribute ignore-if-not-found="true". The data-source in this case will look like if the content was not requested at all.

You can fetch content from any REST API and use it as a data-source in <OnestopEndpoints/>.

The data-source command to fetch xml or json data from any URL is <xml-from-url>. JSON or HTML returned to this command will automatically converted into XML.

For JSON to XML conversion:

• Any characters which would be illegal in XML (for example element name starting with a digit) replaced by _xxxx_ containing their hex unicode character code.

• Note that if any JSON objects have a key _content, then a single XML element is created, with the value of that _content key as the text body, and other keys from the JSON object being attributes on the resulting XML element.

Basic Syntax

<data-source>
    <xml-from-url>
        <!-- mandatory -->
        <url>http://www.google.com/foo/${variable}</url>
        <!-- optional, default GET -->
        <method name="POST"/>
        <!-- zero or more -->
        <get-parameter name="foo">${variable}</get-parameter>
        <!-- zero or more -->
        <request-header name="foo">${variable}</request-header>
        <!-- optional -->
        <basic-access-authentication username="${variable}" password="${variable}"/>
        <!-- either one <xml-body>, or <json-body>, or neither -->
        <json-body>...</json-body>
    </xml-from-url>
<data-source>

<url> is mandatory. That should ne no surprise ;-)

<method> can be "POST" or "GET". If omitted "GET" will be used as a default.

Zero or many <get-parameter>, zero or many <request-header> and zero or one <basic-access-authentication> - all optional.

Request Body

$inline[badge,Highlight,success] The beauty of <OpenEndpoints/> shows in the solution of the optional request body, which can be JSON or XML. There are several different options how-to build the content for the request-body.

XML-Request Body with Inline Contents

The request body is expressed as XML within the <xml-body> tag. Endpoint parameters are expanded.

<xml-from-url>
    ...
    <xml-body>
        <your-tag>${parameter}</your-tag>
    </xml-body>
    ...
</xml-from-url>

Uploaded content encoded in base64 can be filled into any tag of the request body. This requires 2 actions:

• Add attribute upload-files="true" to <xml-from-url>

• Add to any element of your request body attributes upload-field-name="foo" encoding="base64"

The uploaded content will expand into that XML element.

It is also possible to send generated content within a request body:

• Add attribute expand-transformations="true" to <xml-from-url>

• Add to any element of your request body attributes xslt-transformation="foo" encoding="base64"

Adding that attribute to the element indicates that the transformation with that name should be executed (for example, generate a PDF file), and the contents of the resulting file should be placed in this tag. The encoding is always base64, no other encodings are supported.

XML Request Body from Transformation

The request body is generated by XSLT. This leaves maximum flexibility to build different content of the request body depending on endpoint parameter values!

<!-- Data source definition -->
<xml-from-url>
    ...
    <xml-body xslt-file="foo.xslt"/>
    ...
</xml-from-url>

Note that this is a transformation within a transformation. The XSLT takes a <parameters> as its input document; This XSLT does not have access to the results of any other data sources. The reason is, that data sources cannot use data produced by another data source.

• The XSLT file is taken from the http-xslt directory.

• The transformation-input to apply that XSLT has <parameters> as its root tag.

<!-- Input to XSLT processing to build HTTP request body -->
<parameters>
    <parameter name="foo" value="abc"/>
    <parameter name="long" employees="def"/>
    ...
</parameters>

The optional attribute upload-files="true" and expand-transformations="true" may be present, as above.

JSON Request Body with Inline Contents

The request body is expressed as JSON within the <json-body> tag. Endpoint parameters are expanded.

<!-- Data source definition -->
<xml-from-url>
    ...
    <json-body>
        {"json": {
            "key": "${foo}"
                },
          ...
        }
    </json-body>
    ...
</xml-from-url>

Endpoint parameters are expanded within the string context of JSON, that is to say that no concern about escaping is necessary.

Options for expanding base64 content from file upload or generated content is not available for JSON.

JSON Request Body from Transformation

The request body is generated by XSLT. That requires that the result of the transformation is valid JSON.

<!-- Data source definition -->
<xml-from-url>
    ...
    <json-body xslt-file="foo.xslt"/>
    ...
</xml-from-url>

Note that this is a transformation within a transformation. The XSLT takes a <parameters> as its input document, see above "XML Request Body from Transformation" for the format of that block.

Root Tag

By default the root-tag of the generated output is <xml-from-url>. Use the optional tag attribute to generate any different root-tag:

<!-- Data source definition -->
<data-source>
    <xml-from-url tag="my-new-root-tag">
        ...
    </xml-from-url>
<data-source>

Different results may be required for the same endpoint based on certain criteria. It is possible to define any number of success elements such as:

<endpoint name="foo">
    <success if="${language}" equals="en">
        <response-from-static filename="path-to-english-file"/>
    </success>
    <success if="${language}" equals="de">
        <response-from-static filename="path-to-german-file"/>
    </success>
    <success if="${language}" notequals="es">
        <response-from-static filename="path-to-english-file"/>
    </success>
    <success>
        <!-- no condition => "otherwise" -->
        <response-from-static filename="path-to-spanish-file"/>
    </success>
</endpoint>

The conditions are considered in the order they’re written in the file, so put more general “catch-all” items at the bottom and more specific “if...” items at the top.

Parameters may also be used in the "equals" or "notequals" attribute. You could for example create a condition like

<success if="${delivery-address}" notequals="${invoice-address}">

Only if=".." equals=".." and if=".." notequals=".." are available.

If the parameter has a value like foo||bar i.e. created as a result of a request such as ?param=foo&param=bar then the equals=".." will check if any of the values match, and notequals=".." will check that none of the values match the value.

This lists the most recent object keys (filenames) out of the bucket specified in the aws-s3-configuration.xml file (see example-customer for example format.).

“Most recent” means the keys of the objects with the most recent last modified timestamp.

The command looks like:

<aws-s3-keys limit="100">
    <folder>foo/bar</folder> <!-- optional -->
    <match-tag name="foo">bar</match-tag>
    <match-tag name="abc">def</match-tag>
</aws-s3-keys>

and the results look like:

<aws-s3-keys>
    <object key="folder/xyz.xml"/>
    ...
</aws-s3-keys>

This reads a particular object (file) from AWS S3. It is assumed that this object contains XML data. The command looks like:

<aws-s3-object key="folder/foo.xml"/>

and the results look like:

<aws-s3-object key="folder/foo.xml">
    ... file contents (XML) ...
</aws-s3-object>

Literal XML

The data-source command <literal-xml> lets you define xml output directly and "literally" in the data-source definition file.

<!-- Data source definition -->
<data-source>
    <literal-xml>
        <any-tag/>
    </literal-xml>
</data-source>

The root tag <literal-xml> is not included in the data-source xml output. In the example above, the generated xml will be:

<!-- Generated XML -->
<transformation-input>
    <any-tag/>
</transformation-input>

This data-source-type can be perfectly used in combination with parameter placeholders. For example, you can use something like this:

<!-- Data source definition -->
<data-source>
    <literal-xml>
        <any-tag>${foo}</any-tag>
    </literal-xml>
</data-source>

If ${foo} equals "hello world", the data-source output will be:

<!-- Generated XML -->
<transformation-input>
    <any-tag>hello world</any-tag>
</transformation-input>

Note that the contents of must be elements, simply placing text straight under the <literal.xml> element will not work.

Application Introspection

This content-source produces as its output a description of the entire application directory structure (=your configuration).

The generated content has a root-tag <application-introspection> and returns

• <directory name="x"> for any directory

• <file name="x"/> for all XML files. The content of the XML file is included as a child of this tag, except the directory xml-from-application. (Use the <xml-from-application> data source, not <application-introspection> to load content from such files.)

• <file name="x"/> for all non-XML files. In this case the content is not in any way included.

On-Demand Incrementing Number

<OpenEndpoints/> can generate unique auto-increment values and provide them as a data-source. Read On-Demand Incrementing Number for more details.

Basic Syntax

The data-source command <xml-from-database> fetches rows from a database and transforms rows and columns into XML.

Currently only MySQL and PostgreSQL are supported. Other databases would require other client JARs that are not provided in the current version of the software.

<data-source>
    <xml-from-database>
        <jdbc-connection-string><![CDATA[jdbc:postgresql://myserver.com/mydb?user=xxx&password=xxx</jdbc-connection-string]]></jdbc-connection-string>
        <sql>SELECT * FROM mytable WHERE id=?</sql>
        <param>${foo}</param>
    </xml-from-database>
</data-source>

Alternative option using the environment-variable to connect to your local endpoints database - in this example (sql!) fetching request details from the request-log:

<data-source>
    <xml-from-database>
        <jdbc-connection-string from-environment-variable="MY_ENV_VAR"/>
        <sql>SELECT * FROM request_log WHERE request_id=?::uuid</sql>
        <param>${SEARCH_REQUEST_ID}</param>
    </xml-from-database>
</data-source>

<jdbc-connection-string> specifies how to connect to the database to perform the query. This element is mandatory.

• If it is present with no attributes, the body of the tab specifies the JDBC URL. Using a CDATA section is recommended to avoid having to perform XML escaping. Don't forget that username and password values must be URL-escaped.

• If it is has an attribute from-environment-variable="foo" then the environment variable with that name is read and should contain the JDBC URL. Note that endpoints parameters are NOT expanded in the name of the variable name, to prevent an attacker having access to other environment variables.

<sql> should be self-explanatory :-)

• Endpoint parameters are NOT expanded as that would allow SQL injection attacks.

• For PostgreSQL, for non-string parameters, ?::int or ?::uuid it is necessary to cast the string supplied by the endpoint parameter into the right type for PostgreSQL.

Zero or more <param> elements, whose body are the contents of any "?" in the <sql> element. Here, endpoint parameters ARE expanded.

Generated output looks like this:

<!-- Generated output -->
<transformation-input>
    <xml-from-database>
        <row>
            <name-of-column-1>xxx</name-of-column-1>
            <name-of-column-2>xxx</name-of-column-2>
            <name-of-column-3>xxx</name-of-column-3>
        </row>
        <row>
            <name-of-column-1>xxx</name-of-column-1>
            <name-of-column-2>xxx</name-of-column-2>
            <name-of-column-3>xxx</name-of-column-3>
        </row>
    </xml-from-database>
</transformation-input>

By default the root-tag of the generated output is <xml-from-database>. Use the optional tag attribute to generate any different root-tag:

<!-- Data source definition -->
<data-source>
    <xml-from-database tag="my-new-root-tag">
    ...
    </xml-from-database>
</data-source>

This feature can be used to download e.g. Word or Excel files.

Only e.g. DOCX and XLSX etc. are supported; old-style DOC and XLS files are not supported.

The body of the document may contain parameter references such as ${foo}, these are expanded to the parameters available through the endpoint processing.

<endpoint name="foo">
    <success>
        <ooxml-parameter-expansion
                source="foo.docx"
                download-filename="bar-${id}.docx"
        />
    </success>
</endpoint>

The endpoint application directory may contain an ooxml-responses directory and within that any referenced files must be present, for example foo.docx in the example above.

Endpoint parameters can be used as placeholders in your data source. On executing the data source definition any parameter placeholder will be replaced by the respective value of the parameter.

You can only use parameters defined in the endpoints.xml file of your configuration.

<endpoint-folder>
    ...
    <parameter name="foo"/>
    <parameter name="long"/>
    ...
</endpoint-folder>

You can not use

• intermediate parameters (from a task)

• content submitted from a file-upload (or - more generally - any additional content submitted from a multi-part message)

Basic Syntax

If my parameter is called "foo" ...

<parameter name="foo"/>

then I can use as a placeholder:

${foo}

Placeholders in the data-source definition file

Placeholders can be used inside the data-source.xml file.

For example, you can select the specific piece of content loaded from CMS by evaluating a submitted parameter value. Or you could load specific database rows selected for a specific parameter value.

For security or technological reason this does not work in every case. For details please refer to the specific sections:

• Load Data From a Local XML File

• Load Data from any REST-API

• Load Data From Databases

• Additional Useful Data Source Types

Inline Placeholders directly in the content source

On loading content from any of these data source types placeholders will be automatically replaced by its parameter values:

• xml-from-application

• xml-from-url

For example, you can use ${foo} as a placeholder directly in your CMS. On loading data from your CMS, actual values will replace the placeholders.

There is the possibility of adding instructions to output the input/output of the transformation to AWS S3.

<transformer data-source="spheres-fo">
  <write-input-to-aws-s3>
    <folder>foo/bar</folder> <!-- optional -->
    <tag name="foo">bar</tag> 
    <tag name="abc">def</tag> 
  </write-input-to-aws-s3>
  ...
  <write-output-to-aws-s3> 
    ...
  </write-output-to-aws-s3>
</transformer>

This creates objects in the S3 bucket which have the tags as specified, the correct Content-Type, and in addition a tag called "environment" which is either "preview" or "live".

This type of syntax specifies that an endpoint "step-1" is executed and on success another endpoint "step-2" will be called.

The result of the original request (i.e. all parameters) is used as the request to the endpoints being forwarded to:

<endpoint name="step-1">
    <success>
        <forward-to-endpoint endpoint-name="step-2"/>
    </success>
</endpoint>

All parameter values are forwarded to the new endpoint.

System parameters such as user agent, client IP address and file uploads are all available at the endpoint forwarded to. They are inherited to the forwarded endpoint.

It’s possible to chain the execution of any number of endpoints in this manner (e.g. endpoint e1 forwards to e2 which itself forwards to e3). A circular chain of such references is not allowed as the processing of such a chain would never end.

The “redirect” from one endpoint to another happens within the Endpoints software; no redirect is actually sent to the user’s browser.

Request Log Behaviour

Only one “request log” gets written, despite a chain of multiple endpoints being processed. Only the first “parameter transformation input/output” is saved with that “request log” entry, despite each endpoint in the chain potentially having its own parameter transformation.

The beauty of <OpenEndpoints/> is its potential to produce custom content on demand. Practically, that means: Data submitted from a webform (or any other method of submitting data) can directly influence

• what content is loaded from which data sources

• what exactly to transformation of a content shall look like.

Endpoint parameters come from the following sources:

1. Parameters submitted from the originating request to the endpoint including, but not limited to, ?x=y GET parameters

2. The result of a parameter transformation (for example, extracting parameters from arbitrary XML sent as a POST)

3. An "intermediate value" generated by one task and consumed by future items (for example, results from an HTTP request which was sent during the processing)

This data source type will output all available parameter values and make it available for data-source transformation.

<!-- Definition of data source -->
<data-source>
    <parameters/>
</data-source>

The generated content looks like this:

<!-- Data generated by the data source -->
<transformation-input>
    <parameters>
        <parameter name="foo" value="abc"/>
        <parameter name="long" value="def"/>
    </parameters>
</transformation-input>

A special type of parameter value is an uploaded file. For example if you are submitting data from a webform which has <input type="file" name="my-upload"/>, then the presentation of input depends on the type of uploaded content.

If the uploaded content can be parsed as XML, this xml will be available in the data-source:

<!-- Data generated by the data source -->
<transformation-input>
    <parameters>
        <parameter name="foo" value="abc"/>
            <file-upload field-name="bar" upload-filename="foo.xml">
                <xyz>
                    <!-- whatever is in uploaded file -->
                </xyz>
            </file-upload>
        <parameter name="long" value="def"/>
    </parameters>
</transformation-input>

If the uploaded content can not be parsed as XML, the output is:

<!-- Data generated by the data source -->
<transformation-input>
    <parameters>
        <parameter name="foo" value="abc"/>
        <file-upload field-name="photo" upload-filename="photo.jpg"/>
        <parameter name="long" value="def"/>
    </parameters>
</transformation-input>

Intermediate parameters are not regular endpoint parameters, i.e. they are not defined as a <parameter> in endpoints.xml and their value does not come from the original request. Intermediate outputs are generated from tasks that execute during the processing of the request. On forwarding an endpoint to another endpoint those parameters will be made available in the parameters-data-source as well:

<!-- Data generated by the data source -->
<transformation-input>
    <parameters>
        ...
        <intermediate-value name="xyz" value="something"/>
        ...
    </parameters>
</transformation-input>

By default the root-tag of the generated output is <parameters>. Use the optional tag attribute to generate any different root-tag:

<!-- Definition of data source -->
<data-source tag="my-new-root-tag">
    <parameters/>
</data-source>

Sometimes it may be required to insert a unique incremental id into a generated content.

Depending on the specific business use case, the incremental id may be unique “perpetual”, or it may be required to re-start the counter every month or every year.

The command to fetch a new auto-increment value is a data-source.

<data-source>
    <on-demand-incrementing-number type="month"/>
</data-source>

Whenever a transformer has a data-source with that command the auto-increment will be triggered. The type attribute may take the values “perpetual”, “year” or “month”. The numbers are unique within the application.

Creation on-demand

The term “on-demand” refers to the fact the number does not get consumed unless it is requested. If one value (e.g. type=”month” ) is consumed, other values (e.g. type=”year”) are not automatically consumed as well.

The value is only consumed if the endpoints request is successful; if the request is not successful the number is again made available to future requests. The numbers do not have any “holes” or missed-out numbers, so are suitable for use in invoice numbers.

Unique per Request & Data-Source

The same endpoint may contain several different transformers, some of which might call the same data-source. For example, the endpoint may include a task to send an email, and the email-body and some attachment both require the same data-source. In this case - the data source is used twice within the same request - they both see the same number. The new incremental id is created per request, not per use of the (same) data-source. However, if you use 2 different data-sources, both calling an auto-increment, then 2 different ids will be created.

Database

The incremented values will be stored in the database. Changing the values in the database will effect the next generated number.

The parameter-transformation-xslt will use parameter-transformation-input.xml as an input and will generate parameter-transformation-output.xml.

The required output scheme is:

<parameter-transformation-output>
    <parameter name="foo" value="xxx"/>
    <parameter name="fang" value="xxx"/>
    ...
    <parameter name="long" value="xxx"/>
<parameter-transformation-output>

• Each parameter existing in endpoints.xml must be present in the output, except if the parameter has a default-value (which will be applied if the parameter were missing in the output).

• Output of a parameter not existing in endpoints.xml will raise an error.

• If the same parameter appears multiple times, then later values override earlier values.

Raise Custom Error

Optionally an <error> tag can be added. If existing, an error will be raised. The custom error message is taken from this tag. Note that an empty error tag <error/> will also raise an error, but with an empty error message.

<parameter-transformation-output>
    <error>This is my error-message</error>
    <parameter name="foo" value="xxx"/>
    <parameter name="fang" value="xxx"/>
    ...
    <parameter name="long" value="xxx"/>
<parameter-transformation-output>

Offer-Ready is built around the concept of dynamically generated content. But of course parts of the content required to build your endpoints may not require being dynamically generated, because they already exist as some sort of “static” content.

Static content such as images or files may be stored in the application’s static directory.

Image Paths in PDF

Image paths within an XSL-FO file may have an absolute path (to some URI), or a relative path.

• Absolute paths will be "executed" by the PDF reader. Make sure that the path is accessible from your client's device.

• Relative paths will use the application’s static directory as a root directory. Images will be embedded in the generated PDF.

Inputs from request and data from optionally added data sources are automatically placed into a temporary XML called <parameter-transformation-input>.

The custom XSLT ("parameter-transformation-xslt") will be applied to this XML.

OpenEndpoints will automatically insert additional useful tags:

Tags added to <input-from-request>

• all parameters submitted in the originating request, having <parameter name="xxx" value="xxx"/>. This explicitly includes the system parameters hash and environment (if not omitted; otherwise the parameter is not existing).

• endpoint: name of the endpoint that has been called.

• debug-requested: if existing the request had a parameter "debug=true". Otherwise this tag will be omitted.

• <http-header name-lowercase="user-agent">Foo</http-header> This may be present multiple times, or not at all. HTTP Headers are case insensitive so e.g. “User-Agent” and “user-AGENT” are the same header. Therefore these are normalized to all lowercase.

• <cookie name="Session">12345</cookie> This may be present multiple times, or not at all. Cookies on the other hand are case sensitive, so it’s possible to have “Session” and “SESSION” as two different cookies with different values, so these aren’t normalized to lowercase.

• ip-address

Tags added to <input-from-application>

• application: the name of the application

• application-display-name: the display-name of the application, if available from the database.

• git-revision: if the application was published from Git via the Service Portal then this contains the Git hash for the revision. (If the application is deployed in "single application mode" then this tag is omitted.)

• debug-allowed: if existing debugging is set to "allowed" in the database. Otherwise this tag will be omitted.

• secret-key: one separate tag for each secret-key

• random-id-per-endpoint: the database request-log adds a random id per request per application

• base-url: The base-url of the application is taken from an environment variable.

Example: Request sent from web form

Example: Json Payload

The json paylod is converted to xml.

Example: XML Paylod

How-to debug

1. Create the parameter-transformation-tag

3. The generated parameter-transformation-input.xml is available in the request-log.

A data source is a list of zero or many commands which fetch content from different content sources and produce XML. The resulting xml output is the input ("transformation-input") for XSLT to generate output documents.

Sometimes it makes sense to apply one or many intermediate steps to modify the loaded content before it becomes a "transformation-input". Possible reasons for this include

• You might want to reuse the same XSLT to generate an output document for different content sources, but these content-sources do not produce the exactly same structure of input. An intermediate transformation step can be used to "normalize" the input among different content sources.

• A complex transformation might be implement more elegant by splitting it into several subsequent steps.

A data-source-post-processinig.xlt can apply xml-transformations within the data-source object.

How does it work?

In the data-source-post-processing-xslt directory under the application, there are zero or more files, each describing a post-processing transformation step. Each file is a XSLT which expects input-data with a root tag <data-source-post-processing-input> and which shall produce any output xml with a root-tag <data-source-post-processing-output>.

Content loaded from source A:

<data-source-post-processing-input>
    <any-xml/>
</data-source-post-processing-input>

Expected output:

<data-source-post-processing-output>
    <any-xml/>
</data-source-post-processing-output>

You can add zero or many data-source-post-processing.xslt to each content-source of your data-source object. For each content-source post-processing will be executed separately. Multiple steps for the same content-source will be executed subsequently in the order of post-processing-xslt files.

In addition you can apply the same logic for the entire data-source object. In this case all content-sources are loaded as a first stept, and post-processing applies for the collection of all content-sources.

<data-source>
    <xml-from-database>
        ...
        <post-process xslt="file-a.xslt"/>
    </xml-from-database>

    <xml-from-url ignore-if-error="true">
        ...
        <post-process xslt="file-b.xslt"/>
    </xml-from-url>

    <application-introspection>
        <post-process xslt="file-c.xslt"/>
    </application-introspection>

      <!--post-processing applied on the complete set of all content-sources-->
    <post-process xslt="file-X.xslt"/>
    <post-process xslt="file-Y.xslt"/>
    <post-process xslt="file-Z.xslt"/>
</data-source>

Transformer file

In the transformers directory under the application, there are zero or more files, each describing a transformation. Subdirectories are not supported.

The root element of each transformer file is <transformer> .

The root <transformer> element has with a mandatory attribute data-source. The is the name of the data-source without file-extension. For example, if you have a file my-data-source.xml in the data-sources directory, then the correct attribute value is data-source="my-data-source".

The <xslt-file> element is optional. If omitted, the data-source will be returned without XSLT transformation. The name attribute in XSLT file element is mandatory. It is the file-name of an XSLT file including the file-extension. Possible file extensions are ".xslt" or ".xsl".

The optional content-type element sets the mime-type of the generated output. The type attribute is mandatory. The value of this attribute is the mime-type that shall be set. If no content-type were set, heuristics are used by Endpoints to guess an appropriate content type.

Possible applications

Omit transformation

<transformer data-source="[name-of-data-source]"/>

The data-source will be wrapped into a root-tag <transformation-input>.

Note that this is useful for developing and debugging a data-source transformation. Omitting the xslt-file element returns exactly the input, which your xslt will be applied to.

Generate XML

<transformer data-source="[name-of-data-source]">
    <xslt-file name="[path-to-xslt-with-output-type-xml]"/>
    <content-type type="text/xml"/>
</transformer>

Generate HTML

<transformer data-source="[name-of-data-source]">
    <xslt-file name="[path-to-xslt-with-output-type-html]"/>
    <content-type type="text/html"/>
</transformer>

Generate PDF