Golang框架實戰-KisFlow流式計算框架專欄

Golang框架實戰-KisFlow流式計算框架(1)-概述
Golang框架實戰-KisFlow流式計算框架(2)-項目構建/基礎模塊-(上)
Golang框架實戰-KisFlow流式計算框架(3)-項目構建/基礎模塊-(下)
Golang框架實戰-KisFlow流式計算框架(4)-數據流
Golang框架實戰-KisFlow流式計算框架(5)-Function調度
Golang框架實戰-KisFlow流式計算框架(6)-Connector
Golang框架實戰-KisFlow流式計算框架(7)-配置導入與導出
Golang框架實戰-KisFlow流式計算框架(8)-KisFlow Action
Golang框架實戰-KisFlow流式計算框架(9)-Cache/Params 數據緩存與數據參數
Golang框架實戰-KisFlow流式計算框架(10)-Flow多副本
Golang框架實戰-KisFlow流式計算框架(11)-Prometheus Metrics統計
Golang框架實戰-KisFlow流式計算框架(12)-基于反射自適應注冊FaaS形參類型

案例：
KisFlow-Golang流式計算案例(一)快速開始QuickStart
KisFlow-Golang流式計算案例(二)-Flow并流操作
KisFlow-Golang流式計算案例(二)-KisFlow在多協程中的應用

DownLoad kis-flow source

$go get github.com/aceld/kis-flow

《KisFlow開發者文檔》

1. KisFlow快速開始(使用配置文件)

案例源代碼： kis-flow-usage/2-quick_start_with_config at main · aceld/kis-flow-usage

首先我們創建一個項目，項目的文件路徑如下：

項目目錄

├── Makefile
├── conf
│   ├── flow-CalStuAvgScore.yml
│   ├── func-AvgStuScore.yml
│   └── func-PrintStuAvgScore.yml
├── faas_stu_score_avg.go
├── faas_stu_score_avg_print.go
└── main.go

Flow

定義當前的Flow，當前的Flow名稱為："CalStuAvgScore",這是一個計算學生平均分值的數據流。

Config

有關Flow和Function的配置文件如下。

(1) Flow Config

conf/flow-CalStuAvgScore.yml

kistype: flow
status: 1
flow_name: CalStuAvgScore
flows:
 - fname: AvgStuScore
 - fname: PrintStuAvgScore

(2) Function1 Config

conf/func-AvgStuScore.yml

kistype: func
fname: AvgStuScore
fmode: Calculate
source:
 name: 學生學分
 must:
 - stu_id

(3) Function2(Slink) Config

conf/func-PrintStuAvgScore.yml

kistype: func
fname: PrintStuAvgScore
fmode: Expand
source:
 name: 學生學分
 must:
 - stu_id

Main

接下來是主邏輯，主要分成三步驟：

• 加載配置文件，獲取Flow實例;
• 提交數據;
• 運行Flow。

main.go

package main

import (
    "context"
    "fmt"

    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
)

func main() {
    ctx := context.Background()

    // Load Configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)

    }

    // Get the flow
    flow1 := kis.Pool().GetFlow("CalStuAvgScore")
    if flow1 == nil {
        panic("flow1 is nil")

    }

    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":101, "score_1":100, "score_2":90, "score_3":80}`)
    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":102, "score_1":100, "score_2":70, "score_3":60}`)

    // Run the flow
    if err := flow1.Run(ctx); err != nil {
        fmt.Println("err: ", err)

    }

    return
}

Function1

第一個計算流程的實現邏輯如下, AvgStuScoreIn 為輸入數據類型，當前有三個學分，AvgStuScoreOut為輸出數據類型，為平均分值。

faas_stu_score_avg.go

package main
import (
    "context"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"

)
type AvgStuScoreIn struct {
    serialize.DefaultSerialize
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
    Score3 int `json:"score_3"`

}
type AvgStuScoreOut struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`

// AvgStuScore(FaaS) 計算學生平均分
func AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {

    for _, row := range rows {
        out := AvgStuScoreOut{
            StuId:    row.StuId,
            AvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,
        }
        // 提交結果數據
        _ = flow.CommitRow(out)


    }
    return nil
}

Function2

打印的計算邏輯為直接打印數據即可，如下。

faas_stu_score_avg_print.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"

)

type PrintStuAvgScoreIn struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

type PrintStuAvgScoreOut struct {
    serialize.DefaultSerialize
}

func PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {
    for _, row := range rows {
        fmt.Printf("stuid: [%+v], avg score: [%+v]\n", row.StuId, row.AvgScore)

    }
    return nil
}

OutPut

最后運行程序，得到結果如下：

Add KisPool FuncName=AvgStuScore
Add KisPool FuncName=PrintStuAvgScore
Add FlowRouter FlowName=CalStuAvgScore
stuid: [101], avg score: [90]
stuid: [102], avg score: [76.66666666666667]</pre>

2. KisFlow快速開始（使用原生接口，動態配置）

案例源代碼： kis-flow-usage/1-quick_start at main · aceld/kis-flow-usage

項目目錄

├── faas_stu_score_avg.go
├── faas_stu_score_avg_print.go
└── main.go

Flow

Main

main.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/common"
    "github.com/aceld/kis-flow/config"
    "github.com/aceld/kis-flow/flow"
    "github.com/aceld/kis-flow/kis"
)

func main() {
    ctx := context.Background()

    // Create a new flow configuration
    myFlowConfig1 := config.NewFlowConfig("CalStuAvgScore", common.FlowEnable)

    // Create new function configuration
    avgStuScoreConfig := config.NewFuncConfig("AvgStuScore", common.C, nil, nil)
    printStuScoreConfig := config.NewFuncConfig("PrintStuAvgScore", common.E, nil, nil)

    // Create a new flow
    flow1 := flow.NewKisFlow(myFlowConfig1)

    // Link functions to the flow
    _ = flow1.Link(avgStuScoreConfig, nil)
    _ = flow1.Link(printStuScoreConfig, nil)

    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":101, "score_1":100, "score_2":90, "score_3":80}`)
    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":102, "score_1":100, "score_2":70, "score_3":60}`)

    // Run the flow
    if err := flow1.Run(ctx); err != nil {
        fmt.Println("err: ", err)
    }

    return
}

func init() {
    // Register functions
    kis.Pool().FaaS("AvgStuScore", AvgStuScore)
    kis.Pool().FaaS("PrintStuAvgScore", PrintStuAvgScore)
}

Function1

faas_stu_score_avg.go

package main

import (
    "context"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type AvgStuScoreIn struct {
    serialize.DefaultSerialize
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
    Score3 int `json:"score_3"`
}

type AvgStuScoreOut struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

// AvgStuScore(FaaS) 計算學生平均分
func AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {
    for _, row := range rows {

        out := AvgStuScoreOut{
            StuId:    row.StuId,
            AvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,
        }

        // 提交結果數據
        _ = flow.CommitRow(out)
    }

    return nil
}

Function2

faas_stu_score_avg_print.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type PrintStuAvgScoreIn struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

type PrintStuAvgScoreOut struct {
    serialize.DefaultSerialize
}

func PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {

    for _, row := range rows {
        fmt.Printf("stuid: [%+v], avg score: [%+v]\n", row.StuId, row.AvgScore)
    }

    return nil
}

OutPut

Add KisPool FuncName=AvgStuScore
Add KisPool FuncName=PrintStuAvgScore
funcName NewConfig source is nil, funcName = AvgStuScore, use default unNamed Source.
funcName NewConfig source is nil, funcName = PrintStuAvgScore, use default unNamed Source.
stuid: [101], avg score: [90]
stuid: [102], avg score: [76.66666666666667]

作者:劉丹冰Aceld github: https://github.com/aceld
KisFlow開源項目地址：https://github.com/aceld/kis-flow

Golang框架實戰-KisFlow流式計算框架專欄

Golang框架實戰-KisFlow流式計算框架(1)-概述
Golang框架實戰-KisFlow流式計算框架(2)-項目構建/基礎模塊-(上)
Golang框架實戰-KisFlow流式計算框架(3)-項目構建/基礎模塊-(下)
Golang框架實戰-KisFlow流式計算框架(4)-數據流
Golang框架實戰-KisFlow流式計算框架(5)-Function調度
Golang框架實戰-KisFlow流式計算框架(6)-Connector
Golang框架實戰-KisFlow流式計算框架(7)-配置導入與導出
Golang框架實戰-KisFlow流式計算框架(8)-KisFlow Action
Golang框架實戰-KisFlow流式計算框架(10)-Flow多副本
Golang框架實戰-KisFlow流式計算框架(11)-Prometheus Metrics統計
Golang框架實戰-KisFlow流式計算框架(12)-基于反射自適應注冊FaaS形參類型
Golang框架實戰-KisFlow流式計算框架(12)-基于反射自適應注冊FaaS形參類型

案例：
KisFlow-Golang流式計算案例(一)快速開始QuickStart
KisFlow-Golang流式計算案例(二)-Flow并流操作
KisFlow-Golang流式計算案例(二)-KisFlow在多協程中的應用