spark 讀取 json 檔案，並轉換型態為 ml vector

Spark 讀取 json 檔案，並轉換型態為 ml vector

讀取檔案帶入schema
初始型態為 mllib vector
再利用 mllib vector asML 轉換為 ml vector

import org.apache.spark.mllib.linalg.VectorUDT
import org.apache.spark.sql.types.{LongType, StructField, StructType}
val schema = StructType(Seq(
  StructField("post_id", LongType, true),  StructField("text_w2v", new VectorUDT, true)))
val w2vdf = spark.read.schema(schema).json(path)
  .map{ case Row(post_id:Long , text_w2v: org.apache.spark.mllib.linalg.Vector) =>
    (post_id,text_w2v.asML)
  }.toDF("post_id","text_w2v")

參考文件:

1.writing Spark Dataframe to JSON loses format for MLLIB Sparse Vector

2.如何转换org.apache.spark.mllib.linalg.SparseVector至org.apache.spark.ml.linalg.SparseVector？