Wednesday, July 26, 2017

Analysing data with Data Analytics Server

When we talk about WSO2 DAS there are a few important things we need to give focus to. They are, Event receivers, Event Streams, Event Stream definitions and Event Stores.

Events, are units of data, that are received by WSO2 DAS using Event Receivers. Through these Event Receivers WSO2 DAS receives events from different transports in JSON, XML, WSO2 Event formats, etc. formats. There are many different Event receivers available in WSO2 DAS, such as HTTP Event Receivers, SOAP Event Receivers, WSO2Event Event Receivers, etc.

Event Streams are known to be a sequence of events of a particular type. The “type” in this context can be defined as an event stream definition.

An Event Stream definition is sort of a schema which describes in which format the events that comes into WSO2 DAS server should be in. Each Event Stream Definition would have a name, a version and most importantly the type of the data that it expects to be sent in to WSO2 DAS as Events.

Once an event is received by the Event Receiver, it would be checked against the Event Stream definition and be persisted to an Event Store. This is happening through the Data Analytics Layer (DAL) where the events will be stored in the Event store (can be a relational database) as a BLOB which is in human unreadable format. Then these events will be analyzed and processed and the processed information will then again be store in a Process Store (This too can be a relational database) in a BLOB format.
These analyzed data will be decrypted by DAL and presented in a human readable format through the Data Explorer of WSO2 DAS.

When it comes to IS Analytics, whatever the analyzed data that are in the Process Store will be presented through the Analytics Dashboard which is available in WSO2 DAS after the data being decrypted from DAL.

However, the API Manager Analytics are visible from the Store/Publisher portals that are shipped with the API Manager product. However, the API manager related events that are stored in the Process store cannot be read directly from the API Manager dashboards as they are in a encrypted format. Only a DAL can decrypt this information into a human readable format. Because of this, we have introduced a way which is using a method called carbonJDBC where the DAL converts these information that are in the process store and store them in an external relational database. This database is then pointed to, from the API Manager dashboards and you will see API Manager analytics accordingly.

Tuesday, June 13, 2017

How to resolve "Un-recognized attribute 'targetFramework'. Note that attribute names are case-sensitive." in IIS

While trying to configure a SOAP service in IIS, I came across various issues where I had to do many things to resolve them. One of them is the following issue.

Server Error in '/' Application.
Configuration Error
Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

Parser Error Message: Un-recognized attribute 'targetFramework'. Note that attribute names are case-sensitive.

Source Error:

Line 3: 
Line 4:   
Line 5:     
Line 6:   

Line 7:   

Source File: C:\RestService\RestService2\RestService\web.config    Line: 5

Version Information: Microsoft .NET Framework Version:2.0.50727.5420; ASP.NET Version:2.0.50727.5459


To resolve this issue, what you need to do is, check whether you have installed ASP.NET 4 on your Windows instance.
If it is installed, open up a command window, go to the location where .NET 4 is available,

C:\Users\Administrator>cd C:\Windows\Microsoft.NET\Framework\v4.0.30319

Then run the following command

aspnet_regiis -i

E.g.:- C:\Windows\Microsoft.NET\Framework\v4.0.30319>aspnet_regiis -i

Once this is done, open your IIS (type inetmgr in run) and change your Application pool setting to .Net4 (Go to Application Pools -> click on your project -> Right click and select 'Basic Settings')





Sunday, June 11, 2017

Where do we have the public GIT repos for XACML and XACML Mediation Feature?

If you want to do changes to the XACML and XACML mediation features and use it within your product, you can get the code from the below locations.

 XACML
https://github.com/wso2/carbon-identity-framework/tree/master/features/xacml  

XACML Mediation Feature https://github.com/wso2/carbon-mediation/tree/master/components/mediators/entitlement/org.wso2.carbon.identity.entitlement.mediator

Monday, June 5, 2017

Some important grep commands

While working on day to day work, there are many important grep commands that comes in handy. Below are some of these commands.

  • How to find the number of occurrences of a particular text in a file.

grep -o "text to be searchd" nohup.out | wc -l
This blog is yet to be developed and I will add one by one when I come across them.

Sunday, June 4, 2017

How does thread switching happen in WSO2 ESB (Switching between ServerWorker & IO reactor threads)

One of the most important things we need to understand when working with WSO2 ESB is it's threading model. There are two types of threads that does the main functionality within the ESB.

a) SynapseWorker threads
b) IO Reactor Threads

Out of these two threads, the SynapseWorker threads handles whatever the operations related to message mediation. On the other hand, the IO reactor threads are available for event handling such as receiving message headers, receiving message body, etc.

Inside the Synapse engine, the message would go through many classes where transformations that are required are done to the incoming message. The classes involved in this flow are explained in a previous post which I have written.

Once the message mediation is done, it is passed onto the Axis2Engine where it's receive() method is called. Inside this method, the type of the receiver is read from the context. In this example, since we are sending the message to a Proxy service, the receiver would be set to ProxyServiceMessageReceiver. So the message context would be passed on to the receive method of the ProxyServiceMessageReceiver class.
This is the point where the incoming message is passed onto the Synapse Engine.



After the mediation flow completes for the incoming message inside the mediation engine, it would be handed over to the PassThroughHttpSender where the outbound HTTP requests can be sent.

The PassThroughHttpSender implements the TransportSender interface. The initialisation of the PassThroughHttpSender happens at the server startup and an instance of the NHttpClientEventHandler is created by the name TargetHandler.

In addition to this, when the PassThroughHttpSender is initialised, it creates an instance of the DefaultConnectingIOReactor as well.
Along with this an instance of the DeliveryAgent is also formed which allows storing of messages for later delivery.
If a connection is available to send the message it would send the message right away and if not, it would put the message to a message queue to be delivered to the backend whenever a connection is available. This implementation is available inside the submit() method of the DeliveryAgent class.

When the message comes in the PassThroughHttpSender it hits it's invoke() method. After removing unwanted headers sent through the request, it would then check the messageContext for the endpoint value sent in the request message.
If an endpoint is passed in from the request, the submit() method of the DeliveryAgent class is called. If no endpoint is sent, it would call the submitResponse() method.

In this scenario, since we have specified an endpoint value, I would explain the flow which continues from the submit() method of the DeliveryAgent class.



Inside the submit() method, it would add the incoming message to a queue as shown below.



Once the message is added to the queue, it would call the getConnection() method of the TargetConnections class. This would return a connection if already available or won't return anything and will notify the delivery agent when a connection is available.



In a scenario where a connection is available, the connected() method of the TargetHandler is called and also a new TargetContext is created and the Protocol state is set to REQUEST_READY. which means that the connection is at the initial stage and is ready to receive a request.


Next the connected() method of the DeliveryAgent class is invoked where it would check the queue for any messages and pass onto the tryNextMessage() method.



Inside this method, the TargetConext will be updated with the status REQUEST_READY. At the same time, the TargetContext will be set with the messageContext and then the message will be sent to the submitRequest() method.



When the submitRequest() method is called, it would create a TargetRequest and attach it to the TargetContext.


Next is the invocation of the requestReady() method of the TargetHandler class where the HTTP headers are written to the TargetRequest.




Then the outputReady() of the TargetHandler is called;  where the write() method of the TargetRequest is hit.



In this method, it reads the data that was written before to the pipe and writes it to the wire. Once this is done, the Protocol status is also updated to REQUEST_DONE.



There you go! Now you know how the request messages are being passed on from the Worker threads to the IO threads within WSO2 ESB.


Sunday, May 28, 2017

How to send an email with a text that has new lines (\n) in ESB 5.0.0


Assume that we want to send an email with the following text.

Hello world!!!

My Name is Evanthika Amarasiri.

I work for the support team.


I need to send this email in the above format with new lines between each sentence.

How can we make this possible with WSO2 ESB?

So to support this, what you first need is to configure the WSO2 ESB to support email sending. This can be done by following the configuration mentioned in our product documentation.

Once done, start up the ESB server and create a Proxy service with the following configuration.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="EmailSender"
       transports="https http"
       startOnLoad="true">
   <description/>
   <target>
      <endpoint>
         <address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
      </endpoint>
      <inSequence>
         <log/>
         <property name="messageType" value="text/xml" scope="axis2" type="STRING"/>
         <property name="ContentType" value="text/xml" scope="axis2"/>
         <property name="Subject" value="Testing ESB" scope="transport"/>
         <property name="OUT_ONLY" value="true"/>
         <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
         <payloadFactory media-type="xml">
            <format>
               <ns:text xmlns:ns="http://ws.apache.org/commons/ns/payload">$1</ns:text>
            </format>
            <args>
               <arg evaluator="xml"
                    expression="concat('Hello world!!!','&#10;','&#10;', 'My Name is Evanthika Amarasiri.','&#10;','&#10;', 'I work for the support team.')"/>
            </args>
         </payloadFactory>
         <log level="full"/>
         <send>
            <endpoint>
               <address uri="mailto:evanthika@wso2.com"/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <send/>
      </outSequence>
   </target>
</proxy>


Note the below line that is inside the PayloadFactory mediator.

<arg evaluator="xml" expression="concat('Hello world!!!', '&#10;','&#10;', 'My Name is Evanthika Amarasiri.','&#10;','&#10;',  'I work for the support team.')"/>

To support new lines what you need to add is ' ' in between the text you want the new line to be.

Once the above proxy service is deployed, send a request to the proxy service and you should get an email attachment with the below content.

 NOTE: In WSO2 ESB 4.8.1 this cannot be done from the UI and has a known issue. Therefore, as a solution, you need to add the configuration to the physical proxy configuration file which resides under wso2esb-4.8.1/repository/deployment/server/synapse-configs/default/proxy-services
 

Friday, May 26, 2017

Validating JSON arrays when the payload is sent as a query parameter with WSO2 ESB

In my previous post, I explained how JSON payloads can be validated when it's sent as a query parameter. Using the same Synapse configuration without doing any changes, we will see how JSON arrays can be validated by tweaking the JSON schema.

Assume that my requirement is to send a JSON array as a query parameter as shown below.
http://localhost:8280/jsonAPI/jsonapi?jsonPayload=[{"getQuote": {"request": {"symbol": "WSO2"}}},{"getQuote": {"request": {"symbol": "MSFT"}}}]

Create an API using the same configuration which we have used in the previous post.

   <api context="/jsonAPI" name="jsonAPI">
        <resource methods="GET" protocol="http" uri-template="/jsonapi">
            <inSequence>
                <property expression="$url:jsonPayload"
                    name="jsonKeyValue" scope="default" type="STRING"/>
                <payloadFactory media-type="json">
                    <format>$1</format>
                    <args>
                        <arg evaluator="xml" expression="get-property('jsonKeyValue')"/>
                    </args>
                </payloadFactory>
                <validate>
                    <schema key="conf:/schema/StockQuoteSchema.json"/>
                    <on-fail>
                        <payloadFactory media-type="json">
                            <format>{"Error":"$1"}</format>
                            <args>
                                <arg evaluator="xml" expression="$ctx:ERROR_MESSAGE"/>
                            </args>
                        </payloadFactory>
                        <respond/>
                    </on-fail>
                </validate>
                <respond/>
            </inSequence>
        </resource>
    </api>

The StockQuoteSchema.json which you have created under the path conf:/schema/StockQuoteSchema.json should be written in the following format.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "array",
    "items": [{
        "getQuote": {
            "type": "object",
            "properties": {
                "request": {
                    "type": "object",
                    "properties": {
                        "symbol": {
                            "type": "string"
                        }
                    },
                    "required": [
                        "symbol"
                    ]
                }
            },
            "required": [
                "request"
            ]
        }
    }],
    "required": [
        "getQuote"
    ]
}

Note the text marked in blue. In the previous example, when a simple JSON payload was sent, the value of the type was set to object whereas in this scenario, since it's handling JSON arrays, it should be set to array.

On the other hand, since your JSON payload is sending an array, the schema should list elements to be checked inside a block called items with the JSON body wrapped inside square brackets i.e.; [] as highlighted above. 

So once the above configuration is done, and the GET request is sent to the API, you should see the following output if everything goes well.

[
  {
    "getQuote": {
      "request": {
        "symbol": "WSO2"
      }
    }
  },
  {
    "getQuote": {
      "request": {
        "symbol": "MSFT"
      }
    }
  }
]