Posts Tagged ‘golang’
Go plugin file size optimization
Now a days most of my pet projects are developed in golang. Recently I started working on a project to read data from some sensors to do some automations at my farm. The program will be running in a raspberry pi. I designed the system to utilize plugin architecture, this approach allow me to expand the functionality with ease. When I start adding plugins, I realized that the plugin binary size is more than I expected. As we have fast internet connectivity these size will not cause much harm but when the system is deployed in a place where internet connections are really slow then the file size really matters. This blog is about how I managed to reduce the plugin file size.
If one designing a pluggable system then real care should be given to the plugin code base. Design the plugin in such a way to reduce import of packages. Golang plugins are self sufficient like any other go binary. So what ever the packages imported will add size to the plugin. To make thing more clear, I wrote a small application which loads several plugins and print hello world.
Lets write some code
Here is the sample plugin looks like
import "fmt"
type Plugin struct {
}
func (d Plugin) Write() {
fmt.Println("Hello world")
}
var P Plugin
Like above created another plugin to write hello world in German
Build the plugin
go build -buildmode=plugin -o writers/plugins/en.so writers/plugins/en/en.go
go build -buildmode=plugin -o writers/plugins/de.so writers/plugins/de/de.go
Now lets examine the size of the plugin
ls -lh ./writers/plugins/*.so
-rw-r--r-- 1 pi pi 3.3M Apr 25 13:57 ./writers/plugins/de.so
-rw-r--r-- 1 pi pi 3.3M Apr 25 13:57 ./writers/plugins/en.so
As you can see each plugin is 3.3mb.
Build again this time let’s use ldflags
go build -ldflags="-s -w" -buildmode=plugin -o writers/plugins/en.so writers/plugins/en/en.go
Check the size again
ls -lh ./writers/plugins/*.so
-rw-r--r-- 1 pi pi 3.3M Apr 25 14:28 ./writers/plugins/de.so
-rw-r--r-- 1 pi pi 2.4M Apr 25 14:28 ./writers/plugins/en.so
Building with ldflags reduces the size of en.so plugin by 1mb
Lets run upx to this binary
sudo apt-get install upx
chmod +x ./writers/plugins/en.so
upx -9 -k ./writers/plugins/en.so
Check the file size again
ls -lh ./writers/plugins/*.so
-rw-r--r-- 1 pi pi 3.3M Apr 25 14:28 ./writers/plugins/de.so
-rwxr-xr-x 1 pi pi 2.1M Apr 25 14:28 ./writers/plugins/en.so
Running upx reduce the size by 0.2mb
This is the max reduction I can get with different build optimization.
Refactor the code
This is where we need to redesign the plugins and should keep refactoring the code to reduce package imports.
Where this size of plugin comes from? Its the import of fmt package. If I comment fmt.Println and build using ldflags and running upx will reduce the plugin size to 893k
ls -lh ./writers/plugins/*.so
-rw-r--r-- 1 pi pi 3.3M Apr 25 14:49 ./writers/plugins/de.so
-rwxr-xr-x 1 pi pi 893K Apr 25 14:49 ./writers/plugins/en.so
So how to keep the file size optimum and achieve the result we need. Interface comes to our rescue.
Lets create an interface, this is just a sample code and not following any naming conventions here
type Plugger interface {
Print(a ...interface{})
}
Every plugin should now relay on this interface to print hello world. See the refactored en plugin
type Plugin struct {
}
func (d Plugin) Write(plugger writers.Plugger) {
plugger.Print("Hello world")
}
var P Plugin
Here is the method that satisfies Plugger interface. This function should be outside of plugins package
import (
"fmt"
)
type PluginUtil struct {
}
func NewPluginUtils() PluginUtil {
return PluginUtil{}
}
func (p PluginUtil) Print(a ...interface{}) {
fmt.Println(a...)
}
Check the size of plugin again
-lh ./writers/plugins/*.so
-rw-r--r-- 1 pi pi 1.5M Apr 25 15:11 ./writers/plugins/de.so
-rwxr-xr-x 1 pi pi 897K Apr 25 15:11 ./writers/plugins/en.so
Source code: https://github.com/sonyarouje/goplugin
simdb a simple json db in GO
Some days ago I decided to learn GO. GO is pretty easy to learn and could learn syntax and semantics in couple of hours. To completely learn a language I normally write a small app in that language. So in my free time, I rewrote the expense service created in nodejs to GO and is now live and we are using it. This whole exercise allow me to learn GO in detail.
For me GO looks to be a great simple language with static type checking. Seems like I will be using GO for my future RPi projects than nodejs. In RPi more often I use a simple json as a db to store, retrieve and update execution rules, Sensor details, etc. In nodejs I use tingodb, couldn’t find some thing very similar in GO, so decided to write one, and is called simdb, a simple json db.
Using simdb I can persist stuct or retrieve or update or delete them from the json db. The db file created by simdb is a simple json file. Let’s see some of the functions in simdb.
Create a new instance of db
driver, err:=db.New("customer")
Insert a new Customer to db
customer:=Customer { CustID:"CUST1", Name:"sarouje", Address: "address", Contact: Contact { Phone:"45533355", Email:"someone@gmail.com", }, } err=driver.Insert(customer) if(err!=nil){ panic(err) }
Get a Customer
var customerFirst Customer err=driver.Open(Customer{}).Where("custid","=","CUST1").First().AsEntity(&customerFirst) if(err!=nil){ panic(err) }
Update a customer
customerFirst.Name="Sony Arouje" err=driver.Update(customerFirst) if(err!=nil){ panic(err) }
Delete a customer
toDel:=Customer{ CustID:"CUST1", } err=driver.Delete(toDel) if(err!=nil){ panic(err) }
Update and Delete operation uses the ID field of the struct to perform it’s operation.
Let’s see the full code.
package main import ( "github.com/sonyarouje/simdb/db" "fmt" ) type Customer struct { CustID string `json:"custid"` Name string `json:"name"` Address string `json:"address"` Contact Contact } type Contact struct { Phone string `json:"phone"` Email string `json:"email"` } //ID any struct that needs to persist should implement this function defined //in Entity interface. func (c Customer) ID() (jsonField string, value interface{}) { value=c.CustID jsonField="custid" return } func main(){ fmt.Println("starting....") driver, err:=db.New("dbs") if(err!=nil){ panic(err) } customer:=Customer { CustID:"CUST1", Name:"sarouje", Address: "address", Contact: Contact { Phone:"45533355", Email:"someone@gmail.com", }, } //creates a new Customer file inside the directory passed as the //parameter to New(). If the Customer file already exist //then insert operation will add the customer data to the array err=driver.Insert(customer) if(err!=nil){ panic(err) } //GET ALL Customer //opens the customer json file and filter all the customers with name sarouje. //AsEntity takes an address to Customer array and fills the result to it. //we can loop through the customers array and retireve the data. var customers []Customer err=driver.Open(Customer{}).Where("name","=","sarouje").Get().AsEntity(&customers) if(err!=nil){ panic(err) } // fmt.Printf("%#v \n", customers) //GET ONE Customer //First() will return the first record from the results //AsEntity takes the address to Customer variable (not an array pointer) var customerFirst Customer err=driver.Open(Customer{}).Where("custid","=","CUST1").First().AsEntity(&customerFirst) if(err!=nil){ panic(err) } //Update function uses the ID() to get the Id field/value to find the record and update the data. customerFirst.Name="Sony Arouje" err=driver.Update(customerFirst) if(err!=nil){ panic(err) } driver.Open(Customer{}).Where("custid","=","CUST1").First().AsEntity(&customerFirst) fmt.Printf("%#v \n", customerFirst) // Delete toDel:=Customer{ CustID:"CUST1", } err=driver.Delete(toDel) if(err!=nil){ panic(err) } }
TODO
The query syntax in simdb is not really great, I need to find a better approach.
Source Code: https://github.com/sonyarouje/simdb
Sony Arouje 