GitHub/ReAgent
1
0
Fork 0
mirror of https://github.com/facebookresearch/ReAgent.git synced 2026-05-17 12:40:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix oss spark operator

Browse Source 
Summary: The object mapping library (jackson) doesn't give default value when the key is missing from json.

Reviewed By: econti

Differential Revision: D15447137

fbshipit-source-id: f25b818b1f03941f7f393b694a92f0b6025845c3
This commit is contained in:
Kittipat Virochsiri
2019-05-21 22:57:47 -07:00
committed by Facebook Github Bot
parent b3415d8d59
commit 7d2669a5d2
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
preprocessing/src/main/scala/com/facebook/spark/rl/Helper.scala
+2 -1
View File
@@ -43,7 +43,8 @@ object Helper {
val dt = sqlContext.sparkSession.catalog
.listColumns(tableName)
.collect
.filter(column => columnNames.contains(column.name))
// null check is required because jackson doesn't care about default values
.filter(column => columnNames != null && columnNames.contains(column.name))
.map(column => column.name -> column.dataType)
.toMap
assert(dt.size == columnNames.size)